[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260806 03:49:37.386317 732 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.183.62:46101
I20260806 03:49:37.389483 732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 03:49:37.391327 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:49:37.430193 740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:49:37.430294 738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:49:37.430248 743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:49:37.432178 732 server_base.cc:1034] running on GCE node
I20260806 03:49:38.736807 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:49:38.737277 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:49:38.737514 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988178737492 us; error 0 us; skew 500 ppm
I20260806 03:49:38.738965 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:49:38.753278 732 webserver.cc:460] Webserver started at http://127.0.183.62:38477/ using document root <none> and password file <none>
I20260806 03:49:38.755040 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:49:38.755365 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:49:38.756181 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:49:38.764703 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/master-0-root/instance:
uuid: "04a540520ec3416fbf44eb025f39c0c1"
format_stamp: "Formatted at 2026-08-06 03:49:38 on dist-test-slave-kvfs"
I20260806 03:49:38.777772 732 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.006s sys 0.005s
I20260806 03:49:38.786757 750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:38.788867 732 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.005s sys 0.001s
I20260806 03:49:38.789424 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/master-0-root
uuid: "04a540520ec3416fbf44eb025f39c0c1"
format_stamp: "Formatted at 2026-08-06 03:49:38 on dist-test-slave-kvfs"
I20260806 03:49:38.789882 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:49:38.844760 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:49:38.847326 732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 03:49:38.848105 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:49:38.959487 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.62:46101
I20260806 03:49:38.959613 818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.62:46101 every 8 connection(s)
I20260806 03:49:38.973657 819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:39.009886 819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1: Bootstrap starting.
I20260806 03:49:39.019703 819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:39.022656 819 log.cc:824] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1: Log is configured to *not* fsync() on all Append() calls
I20260806 03:49:39.029433 819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1: No bootstrap required, opened a new log
I20260806 03:49:39.059567 819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a540520ec3416fbf44eb025f39c0c1" member_type: VOTER }
I20260806 03:49:39.060597 819 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:39.060964 819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04a540520ec3416fbf44eb025f39c0c1, State: Initialized, Role: FOLLOWER
I20260806 03:49:39.062489 819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a540520ec3416fbf44eb025f39c0c1" member_type: VOTER }
I20260806 03:49:39.063166 819 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:49:39.063576 819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:49:39.064000 819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:39.070554 819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a540520ec3416fbf44eb025f39c0c1" member_type: VOTER }
I20260806 03:49:39.071943 819 leader_election.cc:302] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04a540520ec3416fbf44eb025f39c0c1; no voters:
I20260806 03:49:39.074617 819 leader_election.cc:288] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:49:39.075089 822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:39.078197 822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 1 LEADER]: Becoming Leader. State: Replica: 04a540520ec3416fbf44eb025f39c0c1, State: Running, Role: LEADER
I20260806 03:49:39.079520 822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a540520ec3416fbf44eb025f39c0c1" member_type: VOTER }
I20260806 03:49:39.081014 819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:49:39.095775 823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04a540520ec3416fbf44eb025f39c0c1. Latest consensus state: current_term: 1 leader_uuid: "04a540520ec3416fbf44eb025f39c0c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a540520ec3416fbf44eb025f39c0c1" member_type: VOTER } }
I20260806 03:49:39.097350 823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [sys.catalog]: This master's current role is: LEADER
I20260806 03:49:39.098153 824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04a540520ec3416fbf44eb025f39c0c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04a540520ec3416fbf44eb025f39c0c1" member_type: VOTER } }
I20260806 03:49:39.099885 824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [sys.catalog]: This master's current role is: LEADER
I20260806 03:49:39.101823 834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:49:39.119032 834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:49:39.122000 732 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 03:49:39.146171 834 catalog_manager.cc:1347] Generated new cluster ID: 36fe57059fb145cfa475af89263cbdbe
I20260806 03:49:39.146596 834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:49:39.181121 834 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:49:39.184433 834 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:49:39.208292 834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1: Generated new TSK 0
I20260806 03:49:39.209939 834 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:49:39.255694 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:49:39.271806 843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:49:39.273188 844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:49:39.279121 847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:49:39.279454 732 server_base.cc:1034] running on GCE node
I20260806 03:49:39.280476 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:49:39.280690 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:49:39.280944 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988179280917 us; error 0 us; skew 500 ppm
I20260806 03:49:39.281677 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:49:39.287743 732 webserver.cc:460] Webserver started at http://127.0.183.1:36917/ using document root <none> and password file <none>
I20260806 03:49:39.288527 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:49:39.288966 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:49:39.289350 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:49:39.290782 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-0-root/instance:
uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
format_stamp: "Formatted at 2026-08-06 03:49:39 on dist-test-slave-kvfs"
I20260806 03:49:39.298194 732 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260806 03:49:39.304128 855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:39.305208 732 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 03:49:39.305567 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-0-root
uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
format_stamp: "Formatted at 2026-08-06 03:49:39 on dist-test-slave-kvfs"
I20260806 03:49:39.305927 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:49:39.325093 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:49:39.326792 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:49:39.331792 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:49:39.337935 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:49:39.338248 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:39.338627 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:49:39.338790 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:39.482887 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.1:42229
I20260806 03:49:39.483085 939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.1:42229 every 8 connection(s)
I20260806 03:49:39.498785 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:49:39.521345 944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:49:39.537324 945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:49:39.542459 947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:49:39.546973 732 server_base.cc:1034] running on GCE node
I20260806 03:49:39.548192 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:49:39.548506 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:49:39.548738 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988179548721 us; error 0 us; skew 500 ppm
I20260806 03:49:39.549528 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:49:39.555131 732 webserver.cc:460] Webserver started at http://127.0.183.2:46479/ using document root <none> and password file <none>
I20260806 03:49:39.555814 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:49:39.556103 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:49:39.556490 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:49:39.558720 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-1-root/instance:
uuid: "24b4fc42da314ef6a58a0b558d70bd22"
format_stamp: "Formatted at 2026-08-06 03:49:39 on dist-test-slave-kvfs"
I20260806 03:49:39.561919 940 heartbeater.cc:347] Connected to a master server at 127.0.183.62:46101
I20260806 03:49:39.562667 940 heartbeater.cc:464] Registering TS with master...
I20260806 03:49:39.565744 940 heartbeater.cc:511] Master 127.0.183.62:46101 requested a full tablet report, sending...
I20260806 03:49:39.569666 732 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.004s sys 0.007s
I20260806 03:49:39.574319 775 ts_manager.cc:194] Registered new tserver with Master: fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229)
I20260806 03:49:39.577178 954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:39.578310 732 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260806 03:49:39.578806 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-1-root
uuid: "24b4fc42da314ef6a58a0b558d70bd22"
format_stamp: "Formatted at 2026-08-06 03:49:39 on dist-test-slave-kvfs"
I20260806 03:49:39.579175 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:49:39.581820 775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41640
I20260806 03:49:39.613377 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:49:39.616008 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:49:39.618198 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:49:39.620903 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:49:39.621137 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:39.621414 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:49:39.621598 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:39.751287 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.2:34259
I20260806 03:49:39.751988 1039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.2:34259 every 8 connection(s)
I20260806 03:49:39.758698 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:49:39.772061 1047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:49:39.774221 1046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:49:39.778076 1049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:49:39.779524 1042 heartbeater.cc:347] Connected to a master server at 127.0.183.62:46101
I20260806 03:49:39.779946 1042 heartbeater.cc:464] Registering TS with master...
I20260806 03:49:39.780880 1042 heartbeater.cc:511] Master 127.0.183.62:46101 requested a full tablet report, sending...
I20260806 03:49:39.781141 732 server_base.cc:1034] running on GCE node
I20260806 03:49:39.782655 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:49:39.782934 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:49:39.783140 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988179783127 us; error 0 us; skew 500 ppm
I20260806 03:49:39.783488 775 ts_manager.cc:194] Registered new tserver with Master: 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:39.783911 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:49:39.785686 775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41644
I20260806 03:49:39.787477 732 webserver.cc:460] Webserver started at http://127.0.183.3:44081/ using document root <none> and password file <none>
I20260806 03:49:39.788321 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:49:39.788645 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:49:39.789008 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:49:39.790704 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-2-root/instance:
uuid: "72263d79dc214f91b6d58264f83e005a"
format_stamp: "Formatted at 2026-08-06 03:49:39 on dist-test-slave-kvfs"
I20260806 03:49:39.797995 732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260806 03:49:39.803393 1056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:39.804590 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 03:49:39.804975 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-2-root
uuid: "72263d79dc214f91b6d58264f83e005a"
format_stamp: "Formatted at 2026-08-06 03:49:39 on dist-test-slave-kvfs"
I20260806 03:49:39.805363 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785988177119491-732-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:49:39.831903 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:49:39.833443 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:49:39.836176 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:49:39.839912 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:49:39.840243 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 03:49:39.840518 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:49:39.840701 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:49:40.001256 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.3:40803
I20260806 03:49:40.001538 1143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.3:40803 every 8 connection(s)
I20260806 03:49:40.024416 1144 heartbeater.cc:347] Connected to a master server at 127.0.183.62:46101
I20260806 03:49:40.024928 1144 heartbeater.cc:464] Registering TS with master...
I20260806 03:49:40.026005 1144 heartbeater.cc:511] Master 127.0.183.62:46101 requested a full tablet report, sending...
I20260806 03:49:40.028559 775 ts_manager.cc:194] Registered new tserver with Master: 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:40.029310 732 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017600692s
I20260806 03:49:40.030500 775 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41658
I20260806 03:49:40.034135 732 test_util.cc:268] Using random seed: -955563472
I20260806 03:49:40.102806 775 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41660:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:49:40.109563 775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:49:40.232698 1092 tablet_service.cc:1467] Processing CreateTablet for tablet 83229f740c4c49389f6a896927da335c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:49:40.233465 1090 tablet_service.cc:1467] Processing CreateTablet for tablet 0b032f67ab214058a66f3628a92b7b4c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:49:40.235772 1091 tablet_service.cc:1467] Processing CreateTablet for tablet 58f7c3e941924310a11c59889b06f270 (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:49:40.235924 1089 tablet_service.cc:1467] Processing CreateTablet for tablet c0ffe10b5eb94e78bb6e646f9a84a55d (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:49:40.238076 1088 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4bbe791ec042679f2862d80343161c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:49:40.238305 1095 tablet_service.cc:1467] Processing CreateTablet for tablet e02c77493ffe4acbb551c14b3d7a623e (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:49:40.238111 1086 tablet_service.cc:1467] Processing CreateTablet for tablet 1219cc03c0af4a0cad5987382c16a506 (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:49:40.245352 1092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83229f740c4c49389f6a896927da335c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.254447 1085 tablet_service.cc:1467] Processing CreateTablet for tablet 0d84e6173c3c4d85a3e176cd5f33134b (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:49:40.251924 1097 tablet_service.cc:1467] Processing CreateTablet for tablet 3284b43248b64e15954ecbf158399dea (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:49:40.251827 987 tablet_service.cc:1467] Processing CreateTablet for tablet 83229f740c4c49389f6a896927da335c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:49:40.257558 984 tablet_service.cc:1467] Processing CreateTablet for tablet 0b032f67ab214058a66f3628a92b7b4c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:49:40.258430 989 tablet_service.cc:1467] Processing CreateTablet for tablet 3284b43248b64e15954ecbf158399dea (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:49:40.257851 988 tablet_service.cc:1467] Processing CreateTablet for tablet e02c77493ffe4acbb551c14b3d7a623e (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:49:40.260639 985 tablet_service.cc:1467] Processing CreateTablet for tablet 58f7c3e941924310a11c59889b06f270 (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:49:40.262951 982 tablet_service.cc:1467] Processing CreateTablet for tablet c0ffe10b5eb94e78bb6e646f9a84a55d (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:49:40.266741 987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83229f740c4c49389f6a896927da335c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.270123 980 tablet_service.cc:1467] Processing CreateTablet for tablet 1219cc03c0af4a0cad5987382c16a506 (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:49:40.272240 979 tablet_service.cc:1467] Processing CreateTablet for tablet 0d84e6173c3c4d85a3e176cd5f33134b (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:49:40.275364 981 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4bbe791ec042679f2862d80343161c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:49:40.277626 1095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02c77493ffe4acbb551c14b3d7a623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.279119 1086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1219cc03c0af4a0cad5987382c16a506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.281845 1085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d84e6173c3c4d85a3e176cd5f33134b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.282516 1090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b032f67ab214058a66f3628a92b7b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.283277 988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02c77493ffe4acbb551c14b3d7a623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.283528 1091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f7c3e941924310a11c59889b06f270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.290982 1097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3284b43248b64e15954ecbf158399dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.294116 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4bbe791ec042679f2862d80343161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.295567 1089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ffe10b5eb94e78bb6e646f9a84a55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.300915 984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b032f67ab214058a66f3628a92b7b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.309185 989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3284b43248b64e15954ecbf158399dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.316154 982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ffe10b5eb94e78bb6e646f9a84a55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.337438 985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f7c3e941924310a11c59889b06f270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.339644 980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1219cc03c0af4a0cad5987382c16a506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.345541 981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4bbe791ec042679f2862d80343161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.348133 979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d84e6173c3c4d85a3e176cd5f33134b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.437194 1165 tablet_bootstrap.cc:492] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:40.439640 893 tablet_service.cc:1467] Processing CreateTablet for tablet e02c77493ffe4acbb551c14b3d7a623e (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260806 03:49:40.452222 889 tablet_service.cc:1467] Processing CreateTablet for tablet c0ffe10b5eb94e78bb6e646f9a84a55d (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260806 03:49:40.438694 894 tablet_service.cc:1467] Processing CreateTablet for tablet 3284b43248b64e15954ecbf158399dea (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260806 03:49:40.445271 892 tablet_service.cc:1467] Processing CreateTablet for tablet 83229f740c4c49389f6a896927da335c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260806 03:49:40.446990 891 tablet_service.cc:1467] Processing CreateTablet for tablet 58f7c3e941924310a11c59889b06f270 (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260806 03:49:40.462617 893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e02c77493ffe4acbb551c14b3d7a623e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.459229 890 tablet_service.cc:1467] Processing CreateTablet for tablet 0b032f67ab214058a66f3628a92b7b4c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260806 03:49:40.465380 885 tablet_service.cc:1467] Processing CreateTablet for tablet 1219cc03c0af4a0cad5987382c16a506 (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260806 03:49:40.472510 892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83229f740c4c49389f6a896927da335c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.465332 1166 tablet_bootstrap.cc:492] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
I20260806 03:49:40.465615 886 tablet_service.cc:1467] Processing CreateTablet for tablet 3d4bbe791ec042679f2862d80343161c (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260806 03:49:40.476397 891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f7c3e941924310a11c59889b06f270. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.477157 889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0ffe10b5eb94e78bb6e646f9a84a55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.482025 883 tablet_service.cc:1467] Processing CreateTablet for tablet 0d84e6173c3c4d85a3e176cd5f33134b (DEFAULT_TABLE table=test-workload [id=be7b9e615b94400b98843491fdafc5eb]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260806 03:49:40.488454 894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3284b43248b64e15954ecbf158399dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.490437 883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d84e6173c3c4d85a3e176cd5f33134b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.493952 885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1219cc03c0af4a0cad5987382c16a506. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.518697 890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b032f67ab214058a66f3628a92b7b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.551295 886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d4bbe791ec042679f2862d80343161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:49:40.577930 1165 tablet_bootstrap.cc:654] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.592315 940 heartbeater.cc:503] Master 127.0.183.62:46101 was elected leader, sending a full tablet report...
I20260806 03:49:40.593657 1167 tablet_bootstrap.cc:492] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:40.604878 1166 tablet_bootstrap.cc:654] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.627435 1167 tablet_bootstrap.cc:654] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.634536 1165 tablet_bootstrap.cc:492] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:40.635067 1165 ts_tablet_manager.cc:1397] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.216s user 0.018s sys 0.005s
I20260806 03:49:40.635285 1166 tablet_bootstrap.cc:492] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:40.635778 1166 ts_tablet_manager.cc:1397] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.191s user 0.006s sys 0.016s
I20260806 03:49:40.638182 1165 raft_consensus.cc:348] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.638896 1165 raft_consensus.cc:374] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:40.638691 1166 raft_consensus.cc:348] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.639257 1165 raft_consensus.cc:729] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:40.639504 1166 raft_consensus.cc:374] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:40.640254 1166 raft_consensus.cc:729] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:40.640621 1165 consensus_queue.cc:260] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.644308 1166 consensus_queue.cc:260] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.650352 1167 tablet_bootstrap.cc:492] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:40.651114 1167 ts_tablet_manager.cc:1397] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.060s user 0.009s sys 0.011s
I20260806 03:49:40.651461 1165 ts_tablet_manager.cc:1428] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.016s user 0.004s sys 0.007s
I20260806 03:49:40.652850 1144 heartbeater.cc:503] Master 127.0.183.62:46101 was elected leader, sending a full tablet report...
I20260806 03:49:40.653143 1165 tablet_bootstrap.cc:492] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:40.654088 1166 ts_tablet_manager.cc:1428] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.018s user 0.004s sys 0.006s
I20260806 03:49:40.654919 1042 heartbeater.cc:503] Master 127.0.183.62:46101 was elected leader, sending a full tablet report...
I20260806 03:49:40.655175 1166 tablet_bootstrap.cc:492] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
I20260806 03:49:40.661029 1165 tablet_bootstrap.cc:654] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.660449 1167 raft_consensus.cc:348] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.661847 1167 raft_consensus.cc:374] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:40.662334 1167 raft_consensus.cc:729] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:40.663424 1167 consensus_queue.cc:260] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.666970 1173 raft_consensus.cc:482] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:40.668457 1173 raft_consensus.cc:504] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.697918 1166 tablet_bootstrap.cc:654] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.701551 1167 ts_tablet_manager.cc:1428] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.050s user 0.008s sys 0.005s
I20260806 03:49:40.705188 1167 tablet_bootstrap.cc:492] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:40.720976 1171 raft_consensus.cc:482] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:40.721611 1171 raft_consensus.cc:504] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.710603 1173 leader_election.cc:288] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
W20260806 03:49:40.758513 941 tablet.cc:2369] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:49:40.767118 1167 tablet_bootstrap.cc:654] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.775426 1171 leader_election.cc:288] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:40.775962 1165 tablet_bootstrap.cc:492] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:40.785401 1165 ts_tablet_manager.cc:1397] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.133s user 0.029s sys 0.027s
I20260806 03:49:40.777083 1166 tablet_bootstrap.cc:492] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:40.790845 1166 ts_tablet_manager.cc:1397] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.136s user 0.020s sys 0.002s
I20260806 03:49:40.789568 1165 raft_consensus.cc:348] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.795173 1165 raft_consensus.cc:374] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:40.795185 1166 raft_consensus.cc:348] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:40.800472 1166 raft_consensus.cc:374] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:40.800784 1166 raft_consensus.cc:729] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:40.801510 1166 consensus_queue.cc:260] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:40.799239 1165 raft_consensus.cc:729] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:40.807646 1167 tablet_bootstrap.cc:492] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:40.808478 1175 raft_consensus.cc:482] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:40.809008 1167 ts_tablet_manager.cc:1397] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.104s user 0.015s sys 0.004s
I20260806 03:49:40.809026 1175 raft_consensus.cc:504] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.807415 1165 consensus_queue.cc:260] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.818399 1175 leader_election.cc:288] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:40.817225 1166 ts_tablet_manager.cc:1428] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.026s user 0.005s sys 0.004s
I20260806 03:49:40.833961 1165 ts_tablet_manager.cc:1428] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.048s user 0.009s sys 0.002s
I20260806 03:49:40.836302 1165 tablet_bootstrap.cc:492] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:40.835139 1166 tablet_bootstrap.cc:492] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
I20260806 03:49:40.864480 1167 raft_consensus.cc:348] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.867009 1167 raft_consensus.cc:374] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:40.867394 1167 raft_consensus.cc:729] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:40.871285 1167 consensus_queue.cc:260] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.845964 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:40.883216 1110 raft_consensus.cc:2454] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 0.
I20260806 03:49:40.887930 958 leader_election.cc:302] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:40.889974 1173 raft_consensus.cc:2781] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:40.890625 1173 raft_consensus.cc:482] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:40.890987 1173 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:40.889745 1167 ts_tablet_manager.cc:1428] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.080s user 0.008s sys 0.003s
I20260806 03:49:40.894369 1165 tablet_bootstrap.cc:654] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.896077 1167 tablet_bootstrap.cc:492] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:40.911998 1166 tablet_bootstrap.cc:654] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.911062 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:40.914184 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
W20260806 03:49:40.918951 1057 leader_election.cc:341] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:49:40.925045 955 leader_election.cc:341] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:40.928532 1173 raft_consensus.cc:504] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.936544 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
W20260806 03:49:40.938413 955 leader_election.cc:341] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:40.941169 1173 leader_election.cc:288] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:40.942633 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:40.946369 1110 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:40.946866 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
W20260806 03:49:40.953181 859 leader_election.cc:341] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:40.955639 1165 tablet_bootstrap.cc:492] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:40.956255 1165 ts_tablet_manager.cc:1397] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.120s user 0.016s sys 0.015s
I20260806 03:49:40.957412 1167 tablet_bootstrap.cc:654] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:40.964553 1165 raft_consensus.cc:348] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.966830 1165 raft_consensus.cc:374] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:40.967582 1165 raft_consensus.cc:729] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:40.972448 1165 consensus_queue.cc:260] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.975396 1165 ts_tablet_manager.cc:1428] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.018s user 0.003s sys 0.005s
I20260806 03:49:40.969158 1110 raft_consensus.cc:2454] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 1.
I20260806 03:49:40.983206 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
I20260806 03:49:40.982782 1165 tablet_bootstrap.cc:492] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:40.985308 1166 tablet_bootstrap.cc:492] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:40.985852 1166 ts_tablet_manager.cc:1397] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.152s user 0.013s sys 0.019s
I20260806 03:49:40.986373 958 leader_election.cc:302] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters: fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:40.986584 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
I20260806 03:49:40.990458 1167 tablet_bootstrap.cc:492] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:40.990959 1167 ts_tablet_manager.cc:1397] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.098s user 0.013s sys 0.009s
W20260806 03:49:40.991812 856 leader_election.cc:341] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:40.992563 856 leader_election.cc:302] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa62c96fa4914471ba169dc0d0dafa9a; no voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:40.996255 1175 raft_consensus.cc:2726] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:40.997354 1167 raft_consensus.cc:348] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:40.998442 1167 raft_consensus.cc:374] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:40.999048 1003 raft_consensus.cc:2379] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72263d79dc214f91b6d58264f83e005a in current term 1: Already voted for candidate 24b4fc42da314ef6a58a0b558d70bd22 in this term.
I20260806 03:49:40.998884 1167 raft_consensus.cc:729] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.001056 1057 leader_election.cc:302] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a; no voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:41.002470 1171 raft_consensus.cc:2726] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.000248 1173 raft_consensus.cc:2781] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.003901 1173 raft_consensus.cc:686] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:41.000168 1167 consensus_queue.cc:260] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.004864 1173 consensus_queue.cc:237] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.006705 1167 ts_tablet_manager.cc:1428] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260806 03:49:41.008178 1166 raft_consensus.cc:348] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.009063 1166 raft_consensus.cc:374] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.009513 1166 raft_consensus.cc:729] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.010581 1166 consensus_queue.cc:260] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.017768 1166 ts_tablet_manager.cc:1428] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.031s user 0.004s sys 0.005s
I20260806 03:49:41.019972 1165 tablet_bootstrap.cc:654] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.015590 1167 tablet_bootstrap.cc:492] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:41.023144 1166 tablet_bootstrap.cc:492] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
I20260806 03:49:41.037050 1173 raft_consensus.cc:482] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.037770 1173 raft_consensus.cc:504] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:41.030166 774 catalog_manager.cc:5495] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 1 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:41.040675 1167 tablet_bootstrap.cc:654] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.043303 1165 tablet_bootstrap.cc:492] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:41.045455 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.046054 1165 ts_tablet_manager.cc:1397] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.014s
W20260806 03:49:41.049506 958 leader_election.cc:341] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.051117 1166 tablet_bootstrap.cc:654] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.051147 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
W20260806 03:49:41.052906 955 leader_election.cc:341] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.053395 955 leader_election.cc:302] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22; no voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:41.051169 1173 leader_election.cc:288] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803), fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229)
I20260806 03:49:41.056988 1172 raft_consensus.cc:2726] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.053815 1165 raft_consensus.cc:348] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.062381 1165 raft_consensus.cc:374] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.063073 1165 raft_consensus.cc:729] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.064378 1165 consensus_queue.cc:260] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.075855 1175 raft_consensus.cc:482] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.076853 1175 raft_consensus.cc:504] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.082882 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
W20260806 03:49:41.085237 856 leader_election.cc:341] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.085525 1166 tablet_bootstrap.cc:492] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:41.086182 1166 ts_tablet_manager.cc:1397] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.063s user 0.005s sys 0.018s
I20260806 03:49:41.089275 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.090418 1110 raft_consensus.cc:2454] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 0.
I20260806 03:49:41.090852 1166 raft_consensus.cc:348] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.092056 1166 raft_consensus.cc:374] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.092430 1166 raft_consensus.cc:729] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.092540 859 leader_election.cc:302] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters: 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:41.094190 1166 consensus_queue.cc:260] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.096609 1175 leader_election.cc:288] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.098639 1175 raft_consensus.cc:2781] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.100234 1175 raft_consensus.cc:482] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.102149 1175 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.103188 1166 ts_tablet_manager.cc:1428] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.017s user 0.012s sys 0.001s
I20260806 03:49:41.104687 1166 tablet_bootstrap.cc:492] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
W20260806 03:49:41.112521 833 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:49:41.113056 1165 ts_tablet_manager.cc:1428] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.064s user 0.002s sys 0.010s
I20260806 03:49:41.115993 1167 tablet_bootstrap.cc:492] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:41.117100 1167 ts_tablet_manager.cc:1397] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.104s user 0.043s sys 0.021s
I20260806 03:49:41.118266 1171 raft_consensus.cc:482] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.119417 1165 tablet_bootstrap.cc:492] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:41.119087 1171 raft_consensus.cc:504] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.121384 1171 leader_election.cc:288] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:41.123986 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.125612 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
W20260806 03:49:41.127712 1057 leader_election.cc:341] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.127939 1167 raft_consensus.cc:348] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.128756 1167 raft_consensus.cc:374] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 03:49:41.129046 1057 leader_election.cc:341] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.128955 1175 raft_consensus.cc:504] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.129583 1057 leader_election.cc:302] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a; no voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:41.130893 1171 raft_consensus.cc:2726] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.129076 1167 raft_consensus.cc:729] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.133088 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:41.133287 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:41.134392 1110 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.133217 1167 consensus_queue.cc:260] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
W20260806 03:49:41.135877 856 leader_election.cc:341] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.136960 1167 ts_tablet_manager.cc:1428] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.019s user 0.007s sys 0.002s
I20260806 03:49:41.137955 1167 tablet_bootstrap.cc:492] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:41.133492 1175 leader_election.cc:288] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.146530 1165 tablet_bootstrap.cc:654] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.148926 1175 raft_consensus.cc:482] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.149463 1175 raft_consensus.cc:504] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.156080 1110 raft_consensus.cc:2454] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 1.
I20260806 03:49:41.157784 859 leader_election.cc:302] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters: 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:41.160480 1166 tablet_bootstrap.cc:654] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.160224 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
I20260806 03:49:41.161667 1167 tablet_bootstrap.cc:654] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
W20260806 03:49:41.162082 856 leader_election.cc:341] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.160499 1209 raft_consensus.cc:2781] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.172787 1175 leader_election.cc:288] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.174243 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
W20260806 03:49:41.178049 859 leader_election.cc:341] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.178778 859 leader_election.cc:302] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa62c96fa4914471ba169dc0d0dafa9a; no voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:41.180757 1175 raft_consensus.cc:2726] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.181264 1209 raft_consensus.cc:686] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Becoming Leader. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:41.182770 1165 tablet_bootstrap.cc:492] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:41.183265 1209 consensus_queue.cc:237] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.186039 1167 tablet_bootstrap.cc:492] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:41.188462 1165 ts_tablet_manager.cc:1397] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.069s user 0.009s sys 0.012s
I20260806 03:49:41.188995 1173 raft_consensus.cc:482] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.189528 1173 raft_consensus.cc:504] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.191869 1173 leader_election.cc:288] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.194996 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.194468 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.192268 1165 raft_consensus.cc:348] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
W20260806 03:49:41.202780 958 leader_election.cc:341] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.202886 1165 raft_consensus.cc:374] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.188592 1167 ts_tablet_manager.cc:1397] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.016s
W20260806 03:49:41.203405 955 leader_election.cc:341] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.204627 1166 tablet_bootstrap.cc:492] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:41.205204 1166 ts_tablet_manager.cc:1397] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.101s user 0.017s sys 0.006s
I20260806 03:49:41.207156 1167 raft_consensus.cc:348] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.208267 1167 raft_consensus.cc:374] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.208581 1167 raft_consensus.cc:729] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.209430 1167 consensus_queue.cc:260] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.209949 1166 raft_consensus.cc:348] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.211026 1166 raft_consensus.cc:374] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.211316 1166 raft_consensus.cc:729] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.211869 1167 ts_tablet_manager.cc:1428] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260806 03:49:41.212039 1166 consensus_queue.cc:260] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.212927 1167 tablet_bootstrap.cc:492] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:41.213213 1165 raft_consensus.cc:729] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.215389 1165 consensus_queue.cc:260] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:41.216686 1166 ts_tablet_manager.cc:1428] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260806 03:49:41.217685 1166 tablet_bootstrap.cc:492] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
I20260806 03:49:41.212738 774 catalog_manager.cc:5495] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a reported cstate change: term changed from 0 to 1, leader changed from <none> to fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1). New cstate: current_term: 1 leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:41.221230 1165 ts_tablet_manager.cc:1428] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.032s user 0.005s sys 0.004s
I20260806 03:49:41.221443 955 leader_election.cc:302] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22; no voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:41.226285 1173 raft_consensus.cc:2726] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.230611 1165 tablet_bootstrap.cc:492] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:41.230094 1166 tablet_bootstrap.cc:654] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.233980 1167 tablet_bootstrap.cc:654] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.249696 1209 raft_consensus.cc:482] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.250535 1209 raft_consensus.cc:504] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.252940 1165 tablet_bootstrap.cc:654] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.254508 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
I20260806 03:49:41.258306 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.259181 1110 raft_consensus.cc:2379] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fa62c96fa4914471ba169dc0d0dafa9a in current term 1: Already voted for candidate 24b4fc42da314ef6a58a0b558d70bd22 in this term.
I20260806 03:49:41.261025 859 leader_election.cc:302] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa62c96fa4914471ba169dc0d0dafa9a; no voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:41.262950 1212 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.262559 1209 leader_election.cc:288] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.282150 1165 tablet_bootstrap.cc:492] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:41.282269 1167 tablet_bootstrap.cc:492] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:41.282083 1212 raft_consensus.cc:2726] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.282984 1165 ts_tablet_manager.cc:1397] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.005s
I20260806 03:49:41.283056 1167 ts_tablet_manager.cc:1397] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.070s user 0.021s sys 0.010s
I20260806 03:49:41.288787 1166 tablet_bootstrap.cc:492] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:41.288623 1171 raft_consensus.cc:482] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.288595 1167 raft_consensus.cc:348] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:41.289397 1166 ts_tablet_manager.cc:1397] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.072s user 0.017s sys 0.012s
I20260806 03:49:41.289600 1167 raft_consensus.cc:374] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.289345 1171 raft_consensus.cc:504] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:41.289896 1167 raft_consensus.cc:729] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.291312 1167 consensus_queue.cc:260] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:41.291744 1171 leader_election.cc:288] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229)
I20260806 03:49:41.294173 1167 ts_tablet_manager.cc:1428] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260806 03:49:41.294615 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
I20260806 03:49:41.295147 1167 tablet_bootstrap.cc:492] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:41.295322 1003 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 0.
I20260806 03:49:41.296494 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.297171 907 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 0.
I20260806 03:49:41.298882 1057 leader_election.cc:302] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:41.300936 1171 raft_consensus.cc:2781] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.301415 1171 raft_consensus.cc:482] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.301766 1171 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.304211 1167 tablet_bootstrap.cc:654] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.288254 1165 raft_consensus.cc:348] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.311335 1171 raft_consensus.cc:504] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:41.312904 1166 raft_consensus.cc:348] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.314096 1166 raft_consensus.cc:374] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.314724 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:41.315439 1003 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.314503 1166 raft_consensus.cc:729] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.316749 1166 consensus_queue.cc:260] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.307240 1165 raft_consensus.cc:374] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.318727 1167 tablet_bootstrap.cc:492] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:41.318703 1165 raft_consensus.cc:729] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.319463 1167 ts_tablet_manager.cc:1397] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.005s
I20260806 03:49:41.319924 1165 consensus_queue.cc:260] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.323379 1167 raft_consensus.cc:348] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.324390 1167 raft_consensus.cc:374] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.324703 1167 raft_consensus.cc:729] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.325613 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:41.325537 1167 consensus_queue.cc:260] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.326478 907 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.327978 1166 ts_tablet_manager.cc:1428] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.038s user 0.006s sys 0.004s
I20260806 03:49:41.329154 1166 tablet_bootstrap.cc:492] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
I20260806 03:49:41.329949 1171 leader_election.cc:288] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229)
I20260806 03:49:41.333462 1167 ts_tablet_manager.cc:1428] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20260806 03:49:41.334905 1167 tablet_bootstrap.cc:492] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:41.350479 1165 ts_tablet_manager.cc:1428] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.067s user 0.022s sys 0.003s
I20260806 03:49:41.351871 1165 tablet_bootstrap.cc:492] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:41.352206 907 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 1.
I20260806 03:49:41.352401 1166 tablet_bootstrap.cc:654] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.355556 1167 tablet_bootstrap.cc:654] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.358639 1057 leader_election.cc:302] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:41.358843 1003 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 1.
I20260806 03:49:41.360121 1222 raft_consensus.cc:2781] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.360677 1222 raft_consensus.cc:686] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Becoming Leader. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:41.361759 1222 consensus_queue.cc:237] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:41.363324 1165 tablet_bootstrap.cc:654] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.370399 1173 raft_consensus.cc:482] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.371106 1173 raft_consensus.cc:504] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.375301 1173 leader_election.cc:288] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.376704 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
W20260806 03:49:41.381026 958 leader_election.cc:341] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.383726 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.384732 907 raft_consensus.cc:2454] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 0.
I20260806 03:49:41.385021 1167 tablet_bootstrap.cc:492] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:41.385628 1167 ts_tablet_manager.cc:1397] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.051s user 0.024s sys 0.004s
I20260806 03:49:41.382686 775 catalog_manager.cc:5495] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a reported cstate change: term changed from 0 to 1, leader changed from <none> to 72263d79dc214f91b6d58264f83e005a (127.0.183.3). New cstate: current_term: 1 leader_uuid: "72263d79dc214f91b6d58264f83e005a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:41.387957 955 leader_election.cc:302] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a; no voters: 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:41.390084 1173 raft_consensus.cc:2781] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.390595 1173 raft_consensus.cc:482] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.390936 1173 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.391590 1167 raft_consensus.cc:348] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.392496 1167 raft_consensus.cc:374] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.392758 1167 raft_consensus.cc:729] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.393375 1167 consensus_queue.cc:260] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.395438 1167 ts_tablet_manager.cc:1428] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260806 03:49:41.396577 1167 tablet_bootstrap.cc:492] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a: Bootstrap starting.
I20260806 03:49:41.401542 1165 tablet_bootstrap.cc:492] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:41.402092 1165 ts_tablet_manager.cc:1397] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.050s user 0.012s sys 0.008s
I20260806 03:49:41.403669 1167 tablet_bootstrap.cc:654] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.406200 1165 raft_consensus.cc:348] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.407598 1165 raft_consensus.cc:374] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.407982 1165 raft_consensus.cc:729] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.408898 1165 consensus_queue.cc:260] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.409390 1172 consensus_queue.cc:1035] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:41.416375 1167 tablet_bootstrap.cc:492] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a: No bootstrap required, opened a new log
I20260806 03:49:41.417037 1167 ts_tablet_manager.cc:1397] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.004s
I20260806 03:49:41.420244 1165 ts_tablet_manager.cc:1428] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.018s user 0.010s sys 0.000s
I20260806 03:49:41.421186 1167 raft_consensus.cc:348] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.422086 1167 raft_consensus.cc:374] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.422403 1167 raft_consensus.cc:729] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.423493 1167 consensus_queue.cc:260] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.429817 1172 consensus_queue.cc:1035] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:49:41.432266 1165 tablet_bootstrap.cc:492] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:41.438020 1167 ts_tablet_manager.cc:1428] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a: Time spent starting tablet: real 0.021s user 0.010s sys 0.000s
I20260806 03:49:41.430249 1173 raft_consensus.cc:504] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.442546 1165 tablet_bootstrap.cc:654] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.441476 1166 tablet_bootstrap.cc:492] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:41.445770 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:41.446316 1166 ts_tablet_manager.cc:1397] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.117s user 0.015s sys 0.008s
I20260806 03:49:41.446457 906 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.452908 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a"
W20260806 03:49:41.454758 958 leader_election.cc:341] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.455093 906 raft_consensus.cc:2454] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 1.
I20260806 03:49:41.456264 955 leader_election.cc:302] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a; no voters: 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:41.457258 1172 raft_consensus.cc:2781] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.459771 1222 raft_consensus.cc:482] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.460290 1222 raft_consensus.cc:504] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.462646 1222 leader_election.cc:288] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:41.464388 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.465126 906 raft_consensus.cc:2454] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 0.
I20260806 03:49:41.465014 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
I20260806 03:49:41.466230 1003 raft_consensus.cc:2454] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 0.
I20260806 03:49:41.466627 1057 leader_election.cc:302] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:41.468233 1222 raft_consensus.cc:2781] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.468685 1222 raft_consensus.cc:482] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.469079 1222 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.471858 1171 raft_consensus.cc:482] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.472494 1171 raft_consensus.cc:504] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.474437 1165 tablet_bootstrap.cc:492] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:41.474997 1165 ts_tablet_manager.cc:1397] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.043s user 0.017s sys 0.009s
I20260806 03:49:41.475044 1171 leader_election.cc:288] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:41.478268 1222 raft_consensus.cc:504] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.479890 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.482314 906 raft_consensus.cc:2454] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 0.
I20260806 03:49:41.482945 1222 leader_election.cc:288] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:41.478163 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
I20260806 03:49:41.485446 1003 raft_consensus.cc:2454] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 0.
I20260806 03:49:41.486886 1057 leader_election.cc:302] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:41.486986 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:41.487884 1003 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.488054 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:41.498446 1172 raft_consensus.cc:686] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:41.497529 1165 raft_consensus.cc:348] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.504655 1165 raft_consensus.cc:374] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.505043 1165 raft_consensus.cc:729] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.504843 906 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.505923 1172 consensus_queue.cc:237] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.506614 1165 consensus_queue.cc:260] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.496809 1222 raft_consensus.cc:2781] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.496436 1173 leader_election.cc:288] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.510682 1238 raft_consensus.cc:482] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.511201 1238 raft_consensus.cc:504] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.512005 1165 ts_tablet_manager.cc:1428] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.037s user 0.015s sys 0.009s
I20260806 03:49:41.512990 1165 tablet_bootstrap.cc:492] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a: Bootstrap starting.
I20260806 03:49:41.514370 1238 leader_election.cc:288] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.514947 1238 raft_consensus.cc:482] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.515614 1238 raft_consensus.cc:504] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.525768 906 raft_consensus.cc:2454] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 1.
I20260806 03:49:41.527884 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.517308 1222 raft_consensus.cc:482] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.529722 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.532469 1057 leader_election.cc:302] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:41.533409 1171 raft_consensus.cc:2781] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.532480 905 raft_consensus.cc:2454] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 0.
I20260806 03:49:41.535259 1165 tablet_bootstrap.cc:654] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.547173 955 leader_election.cc:302] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:41.530759 1166 raft_consensus.cc:348] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.553720 1166 raft_consensus.cc:374] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.554109 1166 raft_consensus.cc:729] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.555116 1166 consensus_queue.cc:260] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.532068 1222 raft_consensus.cc:3037] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.561210 1166 ts_tablet_manager.cc:1428] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.115s user 0.004s sys 0.005s
I20260806 03:49:41.563825 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.564987 1107 raft_consensus.cc:2454] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 0.
I20260806 03:49:41.565461 906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.566200 906 raft_consensus.cc:2454] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 0.
I20260806 03:49:41.556924 1173 raft_consensus.cc:2781] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.567155 958 leader_election.cc:302] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:41.558928 1238 leader_election.cc:288] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.575217 1165 tablet_bootstrap.cc:492] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a: No bootstrap required, opened a new log
I20260806 03:49:41.569547 1173 raft_consensus.cc:482] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.579030 1165 ts_tablet_manager.cc:1397] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.015s
I20260806 03:49:41.580607 1212 consensus_queue.cc:1035] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260806 03:49:41.570495 1171 raft_consensus.cc:686] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Becoming Leader. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:41.587594 1166 tablet_bootstrap.cc:492] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
I20260806 03:49:41.579754 1173 raft_consensus.cc:3037] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.587496 1171 consensus_queue.cc:237] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.599071 1222 raft_consensus.cc:504] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.623099 1212 raft_consensus.cc:482] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.624516 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:41.667836 1165 raft_consensus.cc:348] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.668556 907 raft_consensus.cc:3037] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.668805 1165 raft_consensus.cc:374] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.669236 1165 raft_consensus.cc:729] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.642902 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:41.669973 1165 consensus_queue.cc:260] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.667892 1212 raft_consensus.cc:504] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.582530 1233 raft_consensus.cc:2781] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.636758 1110 raft_consensus.cc:2379] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 24b4fc42da314ef6a58a0b558d70bd22 in current term 1: Already voted for candidate 72263d79dc214f91b6d58264f83e005a in this term.
I20260806 03:49:41.675961 1233 raft_consensus.cc:482] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.642426 1173 raft_consensus.cc:504] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.678257 1233 raft_consensus.cc:3037] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.623482 1222 leader_election.cc:288] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:41.676978 773 catalog_manager.cc:5495] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 1 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:41.683269 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:41.684561 1173 leader_election.cc:288] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.687405 1166 tablet_bootstrap.cc:654] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.688548 1165 ts_tablet_manager.cc:1428] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a: Time spent starting tablet: real 0.103s user 0.009s sys 0.003s
I20260806 03:49:41.582163 1003 raft_consensus.cc:2454] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 1.
I20260806 03:49:41.696164 1212 leader_election.cc:288] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.697499 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.698246 1110 raft_consensus.cc:2454] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 0.
I20260806 03:49:41.699791 859 leader_election.cc:302] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:41.700721 1212 raft_consensus.cc:2781] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.701084 1212 raft_consensus.cc:482] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.701424 1212 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.693138 1001 raft_consensus.cc:2379] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 72263d79dc214f91b6d58264f83e005a in current term 1: Already voted for candidate 24b4fc42da314ef6a58a0b558d70bd22 in this term.
I20260806 03:49:41.705387 999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
W20260806 03:49:41.711058 856 leader_election.cc:341] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:49:41.712378 856 consensus_peers.cc:487] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a -> Peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Couldn't send request to peer 24b4fc42da314ef6a58a0b558d70bd22. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:49:41.712428 1212 raft_consensus.cc:504] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.719952 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:41.724947 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:41.725669 1110 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.731088 1233 raft_consensus.cc:504] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.724375 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:41.732740 1107 raft_consensus.cc:2379] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 24b4fc42da314ef6a58a0b558d70bd22 in current term 1: Already voted for candidate 72263d79dc214f91b6d58264f83e005a in this term.
W20260806 03:49:41.723950 856 leader_election.cc:341] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:49:41.734076 958 leader_election.cc:302] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22; no voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:41.735401 1173 raft_consensus.cc:2726] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.735826 907 raft_consensus.cc:2454] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 1.
I20260806 03:49:41.724779 1212 leader_election.cc:288] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.737970 1057 leader_election.cc:302] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters: 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:41.738304 1110 raft_consensus.cc:2454] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 1.
I20260806 03:49:41.738936 1222 raft_consensus.cc:2781] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.739403 1222 raft_consensus.cc:686] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Becoming Leader. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:41.739554 859 leader_election.cc:302] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters: 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:41.740334 1222 consensus_queue.cc:237] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.741379 1212 raft_consensus.cc:2781] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.745385 1212 raft_consensus.cc:686] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Becoming Leader. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:41.756055 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:41.757299 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:41.758548 1110 raft_consensus.cc:3037] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.757745 907 raft_consensus.cc:3037] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.763772 1166 tablet_bootstrap.cc:492] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:41.762838 774 catalog_manager.cc:5495] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a reported cstate change: term changed from 0 to 1, leader changed from <none> to 72263d79dc214f91b6d58264f83e005a (127.0.183.3). New cstate: current_term: 1 leader_uuid: "72263d79dc214f91b6d58264f83e005a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: HEALTHY } } }
I20260806 03:49:41.764339 1166 ts_tablet_manager.cc:1397] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.177s user 0.031s sys 0.003s
I20260806 03:49:41.758002 1233 leader_election.cc:288] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.756167 1212 consensus_queue.cc:237] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.767681 1166 raft_consensus.cc:348] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.770135 1166 raft_consensus.cc:374] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.770460 1166 raft_consensus.cc:729] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.771114 1166 consensus_queue.cc:260] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.773300 1166 ts_tablet_manager.cc:1428] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 03:49:41.774201 1166 tablet_bootstrap.cc:492] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22: Bootstrap starting.
I20260806 03:49:41.779538 907 raft_consensus.cc:2454] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 1.
I20260806 03:49:41.784756 955 leader_election.cc:302] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:41.785082 1110 raft_consensus.cc:2454] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 1.
I20260806 03:49:41.789788 1166 tablet_bootstrap.cc:654] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22: Neither blocks nor log segments found. Creating new log.
I20260806 03:49:41.791131 775 catalog_manager.cc:5495] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a reported cstate change: term changed from 0 to 1, leader changed from <none> to fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1). New cstate: current_term: 1 leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:41.795564 1233 raft_consensus.cc:2781] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.796672 1233 raft_consensus.cc:686] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:41.797966 1233 consensus_queue.cc:237] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.805517 1222 consensus_queue.cc:1035] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:41.820559 1222 raft_consensus.cc:482] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.822902 1222 raft_consensus.cc:504] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.825049 1222 leader_election.cc:288] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:41.826012 775 catalog_manager.cc:5495] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 1 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:41.827375 1001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22" is_pre_election: true
I20260806 03:49:41.829591 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.830575 907 raft_consensus.cc:2379] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 72263d79dc214f91b6d58264f83e005a in current term 1: Already voted for candidate 24b4fc42da314ef6a58a0b558d70bd22 in this term.
I20260806 03:49:41.829511 774 catalog_manager.cc:5495] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a reported cstate change: term changed from 0 to 1, leader changed from <none> to 72263d79dc214f91b6d58264f83e005a (127.0.183.3). New cstate: current_term: 1 leader_uuid: "72263d79dc214f91b6d58264f83e005a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: HEALTHY } } }
I20260806 03:49:41.832504 1057 leader_election.cc:302] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a; no voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:41.833822 1243 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.835490 1166 tablet_bootstrap.cc:492] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22: No bootstrap required, opened a new log
I20260806 03:49:41.835249 1171 consensus_queue.cc:1035] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:49:41.835943 1166 ts_tablet_manager.cc:1397] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22: Time spent bootstrapping tablet: real 0.062s user 0.016s sys 0.016s
I20260806 03:49:41.838979 1166 raft_consensus.cc:348] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.839685 1166 raft_consensus.cc:374] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:49:41.839951 1166 raft_consensus.cc:729] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Initialized, Role: FOLLOWER
I20260806 03:49:41.840622 1166 consensus_queue.cc:260] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.842624 1166 ts_tablet_manager.cc:1428] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 03:49:41.847785 1173 raft_consensus.cc:482] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.848459 1173 raft_consensus.cc:504] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.850683 1233 raft_consensus.cc:482] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.851179 1233 raft_consensus.cc:504] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.852941 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.853446 1233 leader_election.cc:288] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.855671 1173 leader_election.cc:288] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.857029 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.857861 1110 raft_consensus.cc:2454] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 0.
I20260806 03:49:41.857695 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.859161 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.859405 905 raft_consensus.cc:2454] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 0.
I20260806 03:49:41.860594 958 leader_election.cc:302] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:41.861665 1173 raft_consensus.cc:2781] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:49:41.862016 1173 raft_consensus.cc:482] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:49:41.862362 1173 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.865124 1243 raft_consensus.cc:2726] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.894905 1173 raft_consensus.cc:504] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.896571 955 leader_election.cc:302] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22; no voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:41.897578 1233 raft_consensus.cc:2726] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.901010 1233 raft_consensus.cc:482] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:49:41.901523 1233 raft_consensus.cc:504] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.903635 1233 leader_election.cc:288] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.904335 1173 leader_election.cc:288] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:41.905438 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:41.905592 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" is_pre_election: true
I20260806 03:49:41.906185 907 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.911267 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:41.911762 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72263d79dc214f91b6d58264f83e005a" is_pre_election: true
I20260806 03:49:41.912213 1107 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.912678 1110 raft_consensus.cc:2379] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 24b4fc42da314ef6a58a0b558d70bd22 in current term 1: Already voted for candidate fa62c96fa4914471ba169dc0d0dafa9a in this term.
I20260806 03:49:41.913890 958 leader_election.cc:302] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22; no voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:41.919605 907 raft_consensus.cc:2454] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 1.
I20260806 03:49:41.919705 1107 raft_consensus.cc:2454] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 1.
I20260806 03:49:41.921155 955 leader_election.cc:302] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:41.922127 1233 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:41.922528 1173 raft_consensus.cc:2781] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:49:41.923046 1173 raft_consensus.cc:686] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:41.923974 1173 consensus_queue.cc:237] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:41.931079 1233 raft_consensus.cc:2726] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:49:41.936286 775 catalog_manager.cc:5495] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 1 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:41.978144 1243 consensus_queue.cc:1035] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:49:41.995639 1222 consensus_queue.cc:1035] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:41.995754 1233 consensus_queue.cc:1035] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:42.022258 1173 consensus_queue.cc:1035] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:42.069036 1107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3284b43248b64e15954ecbf158399dea"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
mode: GRACEFUL
new_leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:39422
I20260806 03:49:42.069898 1107 raft_consensus.cc:595] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Received request to transfer leadership to TS fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:42.092505 1001 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:52874
I20260806 03:49:42.093158 1001 raft_consensus.cc:595] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Received request to transfer leadership to TS fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:42.098984 1001 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83229f740c4c49389f6a896927da335c"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:52874
I20260806 03:49:42.099557 1001 raft_consensus.cc:595] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Received request to transfer leadership to TS fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:42.107892 1107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
mode: GRACEFUL
new_leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:39422
I20260806 03:49:42.108610 1107 raft_consensus.cc:595] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Received request to transfer leadership to TS 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:42.117945 1107 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
mode: GRACEFUL
new_leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:39422
I20260806 03:49:42.118804 1107 raft_consensus.cc:595] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Received request to transfer leadership to TS 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:42.152436 907 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
mode: GRACEFUL
new_leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:60994
I20260806 03:49:42.153124 907 raft_consensus.cc:595] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Received request to transfer leadership to TS 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:42.159219 1001 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:52874
I20260806 03:49:42.159904 1001 raft_consensus.cc:595] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Received request to transfer leadership to TS 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:42.217229 1222 consensus_queue.cc:1035] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:49:42.226712 1171 raft_consensus.cc:979] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a: : Instructing follower 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:42.227340 1265 raft_consensus.cc:1067] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Signalling peer 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:42.229873 998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:41480
I20260806 03:49:42.230589 998 raft_consensus.cc:482] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:42.230964 998 raft_consensus.cc:3037] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.235905 1265 consensus_queue.cc:1035] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:42.240801 998 raft_consensus.cc:504] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.242223 1003 raft_consensus.cc:1226] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Rejecting Update request from peer 72263d79dc214f91b6d58264f83e005a for earlier term 1. Current term is 2. Ops: []
I20260806 03:49:42.243345 998 leader_election.cc:288] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 2 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:42.244596 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:42.245220 1107 raft_consensus.cc:3032] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:42.245535 1107 raft_consensus.cc:729] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:42.249521 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b032f67ab214058a66f3628a92b7b4c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:42.246434 1107 consensus_queue.cc:260] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.249651 1212 consensus_queue.cc:1035] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:49:42.250509 1107 raft_consensus.cc:3037] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.257912 1001 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:49:42.260231 1173 consensus_queue.cc:1035] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:49:42.263525 1172 raft_consensus.cc:979] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:42.263984 1172 raft_consensus.cc:1067] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Signalling peer fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:42.269814 1107 raft_consensus.cc:2454] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 2.
I20260806 03:49:42.269945 906 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:44436
I20260806 03:49:42.275929 906 raft_consensus.cc:482] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:42.276363 906 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.280292 958 leader_election.cc:302] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters: fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:42.281555 1172 raft_consensus.cc:2781] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:42.282014 1172 raft_consensus.cc:686] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:42.282976 1172 consensus_queue.cc:237] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.291944 906 raft_consensus.cc:504] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
W20260806 03:49:42.252218 1171 consensus_queue.cc:1159] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "24b4fc42da314ef6a58a0b558d70bd22" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 72263d79dc214f91b6d58264f83e005a for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:49:42.302112 1286 raft_consensus.cc:979] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a: : Instructing follower 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:42.307900 1212 raft_consensus.cc:1067] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Signalling peer 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:42.308312 1222 raft_consensus.cc:979] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a: : Instructing follower fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:42.308816 1222 raft_consensus.cc:1067] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Signalling peer fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:42.310423 1003 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:41484
I20260806 03:49:42.308377 906 leader_election.cc:288] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 2 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:42.309900 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:42.312271 907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3284b43248b64e15954ecbf158399dea"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:44446
I20260806 03:49:42.312568 998 raft_consensus.cc:3032] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:42.312824 1110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:42.312996 907 raft_consensus.cc:482] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:42.313838 907 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.314759 1233 consensus_queue.cc:1035] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:49:42.313457 1110 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.318964 774 catalog_manager.cc:5495] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 1 to 2, leader changed from 72263d79dc214f91b6d58264f83e005a (127.0.183.3) to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 2 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:42.320247 1003 raft_consensus.cc:482] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:42.320670 1003 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.312974 998 raft_consensus.cc:729] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:42.324687 998 consensus_queue.cc:260] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.326040 998 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.339288 1110 raft_consensus.cc:2454] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 2.
I20260806 03:49:42.356782 1173 raft_consensus.cc:979] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:42.354864 859 leader_election.cc:302] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:42.357353 1173 raft_consensus.cc:1067] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Signalling peer 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:42.358330 1212 raft_consensus.cc:2781] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:42.358986 1212 raft_consensus.cc:686] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Becoming Leader. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:42.359658 1107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:40864
I20260806 03:49:42.360383 1107 raft_consensus.cc:482] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:42.359912 1212 consensus_queue.cc:237] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.360767 1107 raft_consensus.cc:3037] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.361174 998 raft_consensus.cc:2454] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 2.
I20260806 03:49:42.350813 1003 raft_consensus.cc:504] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.377240 1212 consensus_queue.cc:1035] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:42.379383 771 catalog_manager.cc:5495] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a reported cstate change: term changed from 1 to 2, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1). New cstate: current_term: 2 leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:42.381632 907 raft_consensus.cc:504] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:42.388950 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:42.389663 905 raft_consensus.cc:3032] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:42.389990 905 raft_consensus.cc:729] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:42.391053 905 consensus_queue.cc:260] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.391525 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:42.392284 1108 raft_consensus.cc:3032] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:42.392364 905 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.392608 1108 raft_consensus.cc:729] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:42.383157 1107 raft_consensus.cc:504] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.396175 1003 leader_election.cc:288] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 2 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:42.394301 1108 consensus_queue.cc:260] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:42.400404 1108 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.399950 1105 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:42.408706 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:42.409492 1003 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.397994 907 leader_election.cc:288] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 2 election: Requested vote from peers 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:42.417907 1107 leader_election.cc:288] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 2 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:42.419822 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:42.421892 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d84e6173c3c4d85a3e176cd5f33134b" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:42.423033 907 raft_consensus.cc:3037] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.423163 1173 consensus_queue.cc:1035] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:42.425426 1108 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 2.
I20260806 03:49:42.426960 859 leader_election.cc:302] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:42.420694 998 raft_consensus.cc:3032] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:42.427799 998 raft_consensus.cc:729] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:42.428071 1212 raft_consensus.cc:2781] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:42.429054 998 consensus_queue.cc:260] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.429600 1212 raft_consensus.cc:686] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Becoming Leader. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:42.430009 905 raft_consensus.cc:2454] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 2.
I20260806 03:49:42.437050 1212 consensus_queue.cc:237] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:42.437625 955 leader_election.cc:302] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a; no voters: 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:42.438517 1173 raft_consensus.cc:2781] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:42.439881 1173 raft_consensus.cc:686] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:42.440831 1173 consensus_queue.cc:237] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.446038 907 raft_consensus.cc:2454] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 2.
I20260806 03:49:42.452306 775 catalog_manager.cc:5495] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 1 to 2, leader changed from fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1) to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 2 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:42.454241 1057 leader_election.cc:302] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:42.455423 1222 raft_consensus.cc:2781] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:42.455796 998 raft_consensus.cc:3037] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.455950 1222 raft_consensus.cc:686] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Becoming Leader. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:42.456954 1222 consensus_queue.cc:237] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.452800 1003 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 2.
I20260806 03:49:42.471560 771 catalog_manager.cc:5495] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a reported cstate change: term changed from 1 to 2, leader changed from 72263d79dc214f91b6d58264f83e005a (127.0.183.3) to fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1). New cstate: current_term: 2 leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: HEALTHY } } }
I20260806 03:49:42.476358 1173 consensus_queue.cc:1035] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:49:42.482779 998 raft_consensus.cc:2454] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 2.
I20260806 03:49:42.495057 774 catalog_manager.cc:5495] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a reported cstate change: term changed from 1 to 2, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to 72263d79dc214f91b6d58264f83e005a (127.0.183.3). New cstate: current_term: 2 leader_uuid: "72263d79dc214f91b6d58264f83e005a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: HEALTHY } } }
I20260806 03:49:42.512400 1265 raft_consensus.cc:979] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a: : Instructing follower 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:42.514640 1265 raft_consensus.cc:1067] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Signalling peer 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:42.517768 998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:41480
I20260806 03:49:42.518425 998 raft_consensus.cc:482] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:42.518836 998 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.525108 998 raft_consensus.cc:504] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.526886 998 leader_election.cc:288] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 2 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:42.528326 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:42.528412 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:42.528957 907 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.529235 1107 raft_consensus.cc:3032] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:42.529651 1107 raft_consensus.cc:729] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:42.530623 1107 consensus_queue.cc:260] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.532240 1107 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.535671 907 raft_consensus.cc:2454] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 2.
I20260806 03:49:42.536962 955 leader_election.cc:302] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:42.537765 1173 raft_consensus.cc:2781] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:42.538244 1173 raft_consensus.cc:686] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:42.539335 1173 consensus_queue.cc:237] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.540227 1107 raft_consensus.cc:2454] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 2.
I20260806 03:49:42.547017 774 catalog_manager.cc:5495] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 1 to 2, leader changed from 72263d79dc214f91b6d58264f83e005a (127.0.183.3) to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 2 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:42.622473 1233 raft_consensus.cc:979] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:42.623026 1173 raft_consensus.cc:1067] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Signalling peer fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:42.624820 907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83229f740c4c49389f6a896927da335c"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:44436
I20260806 03:49:42.625474 907 raft_consensus.cc:482] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:42.625887 907 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.633842 907 raft_consensus.cc:504] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.636235 907 leader_election.cc:288] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 2 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:42.637462 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:42.637596 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:42.638139 1107 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.638381 998 raft_consensus.cc:3032] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:42.638814 998 raft_consensus.cc:729] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:42.639806 998 consensus_queue.cc:260] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.640985 998 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.645877 1107 raft_consensus.cc:2454] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 2.
I20260806 03:49:42.646255 998 raft_consensus.cc:2454] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 2.
I20260806 03:49:42.647323 859 leader_election.cc:302] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:42.648320 1212 raft_consensus.cc:2781] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:42.648825 1212 raft_consensus.cc:686] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Becoming Leader. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:42.649716 1212 consensus_queue.cc:237] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:42.658034 774 catalog_manager.cc:5495] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a reported cstate change: term changed from 1 to 2, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1). New cstate: current_term: 2 leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:42.712922 1107 raft_consensus.cc:1261] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.714462 1173 consensus_queue.cc:1035] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:42.717377 907 raft_consensus.cc:3037] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.728273 907 raft_consensus.cc:1261] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.729717 1172 consensus_queue.cc:1035] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:42.799530 998 raft_consensus.cc:1261] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.801118 1212 consensus_queue.cc:1035] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:42.812502 1107 raft_consensus.cc:1261] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.814061 1286 consensus_queue.cc:1035] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:49:42.860208 907 raft_consensus.cc:1261] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.860054 998 raft_consensus.cc:1261] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.862166 1172 consensus_queue.cc:1035] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:42.862229 1222 consensus_queue.cc:1035] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:42.874754 1107 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:42.874928 907 raft_consensus.cc:1261] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.877125 1222 consensus_queue.cc:1035] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260806 03:49:42.887485 1107 raft_consensus.cc:1261] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.889210 1233 consensus_queue.cc:1035] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:49:42.907802 998 raft_consensus.cc:1261] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.909462 1286 consensus_queue.cc:1035] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:42.920291 1107 raft_consensus.cc:1261] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:42.921984 1287 consensus_queue.cc:1035] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:49:43.115020 907 raft_consensus.cc:1261] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:43.116539 1233 consensus_queue.cc:1035] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:43.126003 1107 raft_consensus.cc:1261] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:43.128399 1172 consensus_queue.cc:1035] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:43.198670 1107 raft_consensus.cc:1261] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:43.200668 1287 consensus_queue.cc:1035] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:43.212190 998 raft_consensus.cc:1261] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:43.213992 1212 consensus_queue.cc:1035] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:49:45.190759 732 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260806 03:49:45.191872 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:45.220290 907 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58f7c3e941924310a11c59889b06f270"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
mode: GRACEFUL
new_leader_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:60998
I20260806 03:49:45.220968 907 raft_consensus.cc:595] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Received request to transfer leadership to TS 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:45.235831 998 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:52890
I20260806 03:49:45.236552 998 raft_consensus.cc:595] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Received request to transfer leadership to TS 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:45.237612 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:49:45.238109 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:45.382813 1318 raft_consensus.cc:979] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a: : Instructing follower 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:45.383361 1312 raft_consensus.cc:1067] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Signalling peer 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:45.384881 1107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58f7c3e941924310a11c59889b06f270"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:40872
I20260806 03:49:45.385447 1107 raft_consensus.cc:482] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:45.385773 1107 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:45.391448 1107 raft_consensus.cc:504] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:45.394105 1107 leader_election.cc:288] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 2 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:45.395449 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:45.395404 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:45.396216 907 raft_consensus.cc:3032] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:45.396278 998 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:45.396540 907 raft_consensus.cc:729] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:45.397334 907 consensus_queue.cc:260] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:45.398535 907 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:45.402432 998 raft_consensus.cc:2454] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 2.
I20260806 03:49:45.403748 1057 leader_election.cc:302] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:45.404345 907 raft_consensus.cc:2454] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 2.
I20260806 03:49:45.404735 1310 raft_consensus.cc:2781] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:45.406560 1310 raft_consensus.cc:686] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Becoming Leader. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:45.407577 1310 consensus_queue.cc:237] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:45.422792 771 catalog_manager.cc:5495] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a reported cstate change: term changed from 1 to 2, leader changed from fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1) to 72263d79dc214f91b6d58264f83e005a (127.0.183.3). New cstate: current_term: 2 leader_uuid: "72263d79dc214f91b6d58264f83e005a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: HEALTHY } } }
I20260806 03:49:45.484970 1173 raft_consensus.cc:979] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:45.485448 1313 raft_consensus.cc:1067] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Signalling peer 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:45.487154 1107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:40864
I20260806 03:49:45.487720 1107 raft_consensus.cc:482] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:45.488075 1107 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:45.493570 1107 raft_consensus.cc:504] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:45.495288 1107 leader_election.cc:288] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 2 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:45.496390 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:45.496660 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:45.497084 907 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:45.497222 998 raft_consensus.cc:3032] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:49:45.497602 998 raft_consensus.cc:729] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:45.498441 998 consensus_queue.cc:260] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:45.499773 998 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:49:45.504674 907 raft_consensus.cc:2454] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 2.
I20260806 03:49:45.506038 1057 leader_election.cc:302] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:45.507068 1310 raft_consensus.cc:2781] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:49:45.506989 998 raft_consensus.cc:2454] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 2.
I20260806 03:49:45.507652 1310 raft_consensus.cc:686] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Becoming Leader. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:45.508615 1310 consensus_queue.cc:237] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:45.516145 771 catalog_manager.cc:5495] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a reported cstate change: term changed from 1 to 2, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to 72263d79dc214f91b6d58264f83e005a (127.0.183.3). New cstate: current_term: 2 leader_uuid: "72263d79dc214f91b6d58264f83e005a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: HEALTHY } } }
I20260806 03:49:45.875114 998 raft_consensus.cc:1261] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:45.875910 907 raft_consensus.cc:1261] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:45.876747 1310 consensus_queue.cc:1035] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:49:45.877632 1320 consensus_queue.cc:1035] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:46.025733 998 raft_consensus.cc:1261] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:46.026983 1333 consensus_queue.cc:1035] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:46.035854 907 raft_consensus.cc:1261] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:49:46.037854 1334 consensus_queue.cc:1035] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:49:46.239393 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:46.245775 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:46.246279 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:47.247424 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:47.253517 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:47.254144 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:48.255767 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:48.262074 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:48.262530 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:49.366915 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:49.372898 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:49.373740 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:49:49.524631 1033 debug-util.cc:398] Leaking SignalData structure 0x7b080012a060 after lost signal to thread 735
W20260806 03:49:49.525825 1033 debug-util.cc:398] Leaking SignalData structure 0x7b080012a740 after lost signal to thread 818
W20260806 03:49:49.526788 1033 debug-util.cc:398] Leaking SignalData structure 0x7b0800206bc0 after lost signal to thread 939
W20260806 03:49:49.527627 1033 debug-util.cc:398] Leaking SignalData structure 0x7b08001fa4e0 after lost signal to thread 1039
W20260806 03:49:49.528474 1033 debug-util.cc:398] Leaking SignalData structure 0x7b08001fe400 after lost signal to thread 1143
I20260806 03:49:50.374846 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:50.383881 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:50.384590 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:51.385552 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:51.390774 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:51.391342 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:52.392426 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:52.397362 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:52.397868 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:53.398923 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:53.404716 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:53.405241 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:54.406378 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:54.411487 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:54.412036 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:55.445899 732 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260806 03:49:55.452472 905 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3284b43248b64e15954ecbf158399dea"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
mode: GRACEFUL
new_leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:60994
I20260806 03:49:55.453375 905 raft_consensus.cc:595] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Received request to transfer leadership to TS 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:55.458767 905 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
mode: GRACEFUL
new_leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:60994
I20260806 03:49:55.459358 905 raft_consensus.cc:595] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Received request to transfer leadership to TS 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:55.463868 905 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83229f740c4c49389f6a896927da335c"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
mode: GRACEFUL
new_leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:60994
I20260806 03:49:55.464421 905 raft_consensus.cc:595] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Received request to transfer leadership to TS 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:55.469642 1108 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58f7c3e941924310a11c59889b06f270"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
mode: GRACEFUL
new_leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:39422
I20260806 03:49:55.470256 1108 raft_consensus.cc:595] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Received request to transfer leadership to TS 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:55.476713 1362 raft_consensus.cc:979] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a: : Instructing follower 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.477347 1359 raft_consensus.cc:1067] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Signalling peer 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.479013 1108 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
mode: GRACEFUL
new_leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:39422
I20260806 03:49:55.479247 998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:41484
I20260806 03:49:55.479679 1108 raft_consensus.cc:595] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Received request to transfer leadership to TS 24b4fc42da314ef6a58a0b558d70bd22
I20260806 03:49:55.479816 998 raft_consensus.cc:482] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:55.480365 998 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.486850 998 raft_consensus.cc:504] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.489112 998 leader_election.cc:288] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:55.489847 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:55.490726 905 raft_consensus.cc:3032] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:49:55.490716 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e02c77493ffe4acbb551c14b3d7a623e" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:55.491247 905 raft_consensus.cc:729] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:55.491554 1108 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.492173 905 consensus_queue.cc:260] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.493739 905 raft_consensus.cc:3037] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.499701 1108 raft_consensus.cc:2454] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.501204 958 leader_election.cc:302] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:55.501397 905 raft_consensus.cc:2454] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.502143 1313 raft_consensus.cc:2781] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:49:55.502620 1313 raft_consensus.cc:686] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:55.503500 1313 consensus_queue.cc:237] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.513682 774 catalog_manager.cc:5495] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 2 to 3, leader changed from fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1) to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 3 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:55.737843 1370 raft_consensus.cc:979] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a: : Instructing follower 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.738446 1354 raft_consensus.cc:1067] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Signalling peer 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.740253 998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:41480
I20260806 03:49:55.740835 998 raft_consensus.cc:482] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:55.741150 998 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.746683 998 raft_consensus.cc:504] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.748843 998 leader_election.cc:288] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:55.750138 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:55.750165 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0ffe10b5eb94e78bb6e646f9a84a55d" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:55.751096 905 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.751089 1108 raft_consensus.cc:3032] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:49:55.751679 1108 raft_consensus.cc:729] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:55.752600 1108 consensus_queue.cc:260] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.754405 1108 raft_consensus.cc:3037] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.759508 905 raft_consensus.cc:2454] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.761077 955 leader_election.cc:302] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:55.762092 1313 raft_consensus.cc:2781] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:49:55.762076 1108 raft_consensus.cc:2454] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.762684 1313 raft_consensus.cc:686] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:55.763509 1313 consensus_queue.cc:237] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.770637 774 catalog_manager.cc:5495] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 2 to 3, leader changed from 72263d79dc214f91b6d58264f83e005a (127.0.183.3) to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 3 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:55.873283 1362 raft_consensus.cc:979] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a: : Instructing follower 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.873879 1359 raft_consensus.cc:1067] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Signalling peer 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.875432 998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3284b43248b64e15954ecbf158399dea"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:41484
I20260806 03:49:55.876055 998 raft_consensus.cc:482] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:55.876312 998 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.881438 998 raft_consensus.cc:504] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:55.883145 998 leader_election.cc:288] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Requested vote from peers 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803), fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229)
I20260806 03:49:55.884169 905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:55.884135 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:55.884891 905 raft_consensus.cc:3032] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:49:55.884975 1108 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.885210 905 raft_consensus.cc:729] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:55.885983 905 consensus_queue.cc:260] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:55.887332 905 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.890468 1108 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.891667 958 leader_election.cc:302] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:55.892434 1313 raft_consensus.cc:2781] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:49:55.892581 905 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.892804 1313 raft_consensus.cc:686] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:55.894034 1313 consensus_queue.cc:237] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:55.902369 774 catalog_manager.cc:5495] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 2 to 3, leader changed from fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1) to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 3 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:55.911346 1362 raft_consensus.cc:979] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a: : Instructing follower 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.911871 1362 raft_consensus.cc:1067] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Signalling peer 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.913800 998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83229f740c4c49389f6a896927da335c"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:41484
I20260806 03:49:55.914359 998 raft_consensus.cc:482] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:55.914719 998 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.916106 905 raft_consensus.cc:1261] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:55.918044 1313 consensus_queue.cc:1035] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:55.921376 998 raft_consensus.cc:504] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.923082 998 leader_election.cc:288] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:55.923254 1354 raft_consensus.cc:979] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a: : Instructing follower 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.923671 1354 raft_consensus.cc:1067] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Signalling peer 24b4fc42da314ef6a58a0b558d70bd22 to start an election
I20260806 03:49:55.925007 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:55.925830 1108 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.927244 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:55.927739 907 raft_consensus.cc:3032] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:49:55.927974 907 raft_consensus.cc:729] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:55.928869 907 consensus_queue.cc:260] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.930226 907 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.940344 998 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58f7c3e941924310a11c59889b06f270"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
from {username='slave'} at 127.0.0.1:41480
I20260806 03:49:55.941112 998 raft_consensus.cc:482] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:55.941498 998 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.945830 1107 raft_consensus.cc:1261] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:55.947659 1361 consensus_queue.cc:1035] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:49:55.948369 1108 raft_consensus.cc:2454] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.948657 907 raft_consensus.cc:2454] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.949918 958 leader_election.cc:302] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:55.952739 1361 raft_consensus.cc:2781] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:49:55.953831 1361 raft_consensus.cc:686] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:55.954558 1361 consensus_queue.cc:237] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.955607 998 raft_consensus.cc:504] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.959220 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:55.959565 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "24b4fc42da314ef6a58a0b558d70bd22" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:55.960173 907 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.960397 1107 raft_consensus.cc:3032] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:49:55.960827 1107 raft_consensus.cc:729] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:55.961827 1107 consensus_queue.cc:260] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.963479 1107 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:55.968183 998 leader_election.cc:288] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:55.969791 774 catalog_manager.cc:5495] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 2 to 3, leader changed from fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1) to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 3 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:55.973829 1107 raft_consensus.cc:2454] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.975486 958 leader_election.cc:302] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b4fc42da314ef6a58a0b558d70bd22, 72263d79dc214f91b6d58264f83e005a; no voters:
I20260806 03:49:55.976207 907 raft_consensus.cc:2454] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b4fc42da314ef6a58a0b558d70bd22 in term 3.
I20260806 03:49:55.976926 1361 raft_consensus.cc:2781] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:49:55.977428 1361 raft_consensus.cc:686] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Becoming Leader. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:55.978360 1361 consensus_queue.cc:237] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:55.988749 774 catalog_manager.cc:5495] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 reported cstate change: term changed from 2 to 3, leader changed from 72263d79dc214f91b6d58264f83e005a (127.0.183.3) to 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2). New cstate: current_term: 3 leader_uuid: "24b4fc42da314ef6a58a0b558d70bd22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:56.187424 907 raft_consensus.cc:1261] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:56.188902 1361 consensus_queue.cc:1035] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:56.198685 1108 raft_consensus.cc:1261] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:56.200302 1361 consensus_queue.cc:1035] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:56.306841 907 raft_consensus.cc:1261] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:56.308209 1361 consensus_queue.cc:1035] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:56.318305 1107 raft_consensus.cc:1261] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:56.319729 1361 consensus_queue.cc:1035] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:49:56.435220 1107 raft_consensus.cc:1261] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:56.436599 1392 consensus_queue.cc:1035] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:56.449209 907 raft_consensus.cc:1261] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:56.450624 1392 consensus_queue.cc:1035] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:49:56.538844 907 raft_consensus.cc:1261] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:56.540184 1392 consensus_queue.cc:1035] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:56.550297 1107 raft_consensus.cc:1261] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Refusing update from remote peer 24b4fc42da314ef6a58a0b558d70bd22: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:56.551963 1361 consensus_queue.cc:1035] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:58.521819 732 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260806 03:49:58.522706 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:58.530675 998 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:52890
I20260806 03:49:58.531442 998 raft_consensus.cc:595] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Received request to transfer leadership to TS fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:58.534263 998 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3284b43248b64e15954ecbf158399dea"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:52890
I20260806 03:49:58.534698 998 raft_consensus.cc:595] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Received request to transfer leadership to TS 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:58.537312 998 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:52890
I20260806 03:49:58.537957 998 raft_consensus.cc:595] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Received request to transfer leadership to TS fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:58.540664 998 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58f7c3e941924310a11c59889b06f270"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:52890
I20260806 03:49:58.541159 998 raft_consensus.cc:595] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Received request to transfer leadership to TS fa62c96fa4914471ba169dc0d0dafa9a
I20260806 03:49:58.543443 998 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "83229f740c4c49389f6a896927da335c"
dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
mode: GRACEFUL
new_leader_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:52890
I20260806 03:49:58.543902 998 raft_consensus.cc:595] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Received request to transfer leadership to TS 72263d79dc214f91b6d58264f83e005a
I20260806 03:49:58.545058 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 03:49:58.545666 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:49:58.562189 1401 raft_consensus.cc:979] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:58.562721 1401 raft_consensus.cc:1067] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Signalling peer fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:58.564419 905 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58f7c3e941924310a11c59889b06f270"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:44436
I20260806 03:49:58.565060 905 raft_consensus.cc:482] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:58.565536 905 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:58.576478 1361 raft_consensus.cc:979] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:58.577229 1401 raft_consensus.cc:1067] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Signalling peer fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:58.576853 905 raft_consensus.cc:504] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.579654 907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:44436
I20260806 03:49:58.580294 907 raft_consensus.cc:482] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:58.580652 907 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:58.581079 905 leader_election.cc:288] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 4 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:58.582521 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:58.582974 998 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f7c3e941924310a11c59889b06f270" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:58.583400 1107 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:58.583523 998 raft_consensus.cc:3032] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:49:58.583853 998 raft_consensus.cc:729] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:58.584702 998 consensus_queue.cc:260] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.586237 998 raft_consensus.cc:3037] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:58.589525 907 raft_consensus.cc:504] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.590585 1107 raft_consensus.cc:2454] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 4.
I20260806 03:49:58.592089 907 leader_election.cc:288] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 3 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:58.592357 859 leader_election.cc:302] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:58.593464 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:58.594311 1003 raft_consensus.cc:3032] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:49:58.594671 1003 raft_consensus.cc:729] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:58.594918 1402 raft_consensus.cc:2781] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:49:58.595665 998 raft_consensus.cc:2454] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 4.
I20260806 03:49:58.595626 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d4bbe791ec042679f2862d80343161c" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:58.595494 1003 consensus_queue.cc:260] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.596431 1107 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:58.596863 1402 raft_consensus.cc:686] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 4 LEADER]: Becoming Leader. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:58.597126 1003 raft_consensus.cc:3037] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:58.598006 1402 consensus_queue.cc:237] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.604564 1107 raft_consensus.cc:2454] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 3.
I20260806 03:49:58.606338 859 leader_election.cc:302] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:58.607043 1003 raft_consensus.cc:2454] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 3.
I20260806 03:49:58.607877 1402 raft_consensus.cc:2781] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:49:58.608551 1402 raft_consensus.cc:686] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 3 LEADER]: Becoming Leader. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:58.609479 1402 consensus_queue.cc:237] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.610630 771 catalog_manager.cc:5495] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a reported cstate change: term changed from 3 to 4, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1). New cstate: current_term: 4 leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:58.631513 771 catalog_manager.cc:5495] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a reported cstate change: term changed from 2 to 3, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1). New cstate: current_term: 3 leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:58.680399 1361 raft_consensus.cc:979] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:58.680972 1401 raft_consensus.cc:1067] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Signalling peer 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:58.682674 1107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "83229f740c4c49389f6a896927da335c"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:40864
I20260806 03:49:58.683218 1107 raft_consensus.cc:482] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:58.683531 1107 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:58.688398 1107 raft_consensus.cc:504] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.690290 1107 leader_election.cc:288] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 4 election: Requested vote from peers fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229), 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259)
I20260806 03:49:58.691433 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:58.691726 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83229f740c4c49389f6a896927da335c" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:58.692050 907 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:58.692368 1003 raft_consensus.cc:3032] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:49:58.692672 1003 raft_consensus.cc:729] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:58.693367 1003 consensus_queue.cc:260] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.694432 1003 raft_consensus.cc:3037] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:58.697475 907 raft_consensus.cc:2454] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 4.
I20260806 03:49:58.698915 1057 leader_election.cc:302] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:58.699963 1400 raft_consensus.cc:2781] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:49:58.700225 1003 raft_consensus.cc:2454] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 4.
I20260806 03:49:58.701575 1400 raft_consensus.cc:686] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 4 LEADER]: Becoming Leader. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:58.702589 1400 consensus_queue.cc:237] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.712342 771 catalog_manager.cc:5495] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a reported cstate change: term changed from 3 to 4, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to 72263d79dc214f91b6d58264f83e005a (127.0.183.3). New cstate: current_term: 4 leader_uuid: "72263d79dc214f91b6d58264f83e005a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: HEALTHY } } }
I20260806 03:49:58.850339 1361 raft_consensus.cc:979] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:58.850946 1401 raft_consensus.cc:1067] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Signalling peer fa62c96fa4914471ba169dc0d0dafa9a to start an election
I20260806 03:49:58.852713 907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506"
dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
from {username='slave'} at 127.0.0.1:44436
I20260806 03:49:58.853257 907 raft_consensus.cc:482] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:58.853510 907 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:58.858446 907 raft_consensus.cc:504] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.859941 907 leader_election.cc:288] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 3 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), 72263d79dc214f91b6d58264f83e005a (127.0.183.3:40803)
I20260806 03:49:58.861094 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "72263d79dc214f91b6d58264f83e005a"
I20260806 03:49:58.861147 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1219cc03c0af4a0cad5987382c16a506" candidate_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:58.861848 1107 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:58.862005 1003 raft_consensus.cc:3032] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:49:58.862433 1003 raft_consensus.cc:729] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:58.863175 1003 consensus_queue.cc:260] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.864315 1003 raft_consensus.cc:3037] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:49:58.866907 1107 raft_consensus.cc:2454] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 3.
I20260806 03:49:58.867982 859 leader_election.cc:302] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:58.868836 1402 raft_consensus.cc:2781] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:49:58.869344 1402 raft_consensus.cc:686] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 LEADER]: Becoming Leader. State: Replica: fa62c96fa4914471ba169dc0d0dafa9a, State: Running, Role: LEADER
I20260806 03:49:58.870399 1003 raft_consensus.cc:2454] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa62c96fa4914471ba169dc0d0dafa9a in term 3.
I20260806 03:49:58.870291 1402 consensus_queue.cc:237] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } }
I20260806 03:49:58.878718 771 catalog_manager.cc:5495] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a reported cstate change: term changed from 2 to 3, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1). New cstate: current_term: 3 leader_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:59.050815 1361 raft_consensus.cc:979] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22: : Instructing follower 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:59.051347 1401 raft_consensus.cc:1067] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Signalling peer 72263d79dc214f91b6d58264f83e005a to start an election
I20260806 03:49:59.052867 1107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3284b43248b64e15954ecbf158399dea"
dest_uuid: "72263d79dc214f91b6d58264f83e005a"
from {username='slave'} at 127.0.0.1:40864
I20260806 03:49:59.053346 1107 raft_consensus.cc:482] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:49:59.053589 1107 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:59.058943 1107 raft_consensus.cc:504] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:59.061122 1107 leader_election.cc:288] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 4 election: Requested vote from peers 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259), fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229)
I20260806 03:49:59.062321 907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fa62c96fa4914471ba169dc0d0dafa9a"
I20260806 03:49:59.062284 1003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3284b43248b64e15954ecbf158399dea" candidate_uuid: "72263d79dc214f91b6d58264f83e005a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "24b4fc42da314ef6a58a0b558d70bd22"
I20260806 03:49:59.062928 907 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:59.062996 1003 raft_consensus.cc:3032] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:49:59.063357 1003 raft_consensus.cc:729] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 24b4fc42da314ef6a58a0b558d70bd22, State: Running, Role: LEADER
I20260806 03:49:59.064132 1003 consensus_queue.cc:260] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:59.065249 1003 raft_consensus.cc:3037] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:49:59.068940 907 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 4.
I20260806 03:49:59.070029 1057 leader_election.cc:302] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72263d79dc214f91b6d58264f83e005a, fa62c96fa4914471ba169dc0d0dafa9a; no voters:
I20260806 03:49:59.070786 1400 raft_consensus.cc:2781] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:49:59.071301 1400 raft_consensus.cc:686] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 4 LEADER]: Becoming Leader. State: Replica: 72263d79dc214f91b6d58264f83e005a, State: Running, Role: LEADER
I20260806 03:49:59.071796 1003 raft_consensus.cc:2454] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72263d79dc214f91b6d58264f83e005a in term 4.
I20260806 03:49:59.072808 1400 consensus_queue.cc:237] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } }
I20260806 03:49:59.082084 771 catalog_manager.cc:5495] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a reported cstate change: term changed from 3 to 4, leader changed from 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2) to 72263d79dc214f91b6d58264f83e005a (127.0.183.3). New cstate: current_term: 4 leader_uuid: "72263d79dc214f91b6d58264f83e005a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 } health_report { overall_health: UNKNOWN } } }
I20260806 03:49:59.087069 1107 raft_consensus.cc:1261] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:49:59.088646 1402 consensus_queue.cc:1035] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:49:59.115334 1003 raft_consensus.cc:1261] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:49:59.116647 1402 consensus_queue.cc:1035] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:49:59.126874 1003 raft_consensus.cc:1261] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:59.128317 1402 consensus_queue.cc:1035] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:59.138242 1107 raft_consensus.cc:1261] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:59.140388 1402 consensus_queue.cc:1035] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:59.151825 1003 raft_consensus.cc:1261] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:49:59.153231 1400 consensus_queue.cc:1035] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:49:59.163516 907 raft_consensus.cc:1261] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:49:59.165257 1412 consensus_queue.cc:1035] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:49:59.426540 1107 raft_consensus.cc:1261] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:59.427871 1402 consensus_queue.cc:1035] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "72263d79dc214f91b6d58264f83e005a" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 40803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:59.437518 1003 raft_consensus.cc:1261] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Refusing update from remote peer fa62c96fa4914471ba169dc0d0dafa9a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:49:59.439513 1402 consensus_queue.cc:1035] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:49:59.474994 1003 raft_consensus.cc:1261] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:49:59.476405 1433 consensus_queue.cc:1035] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b4fc42da314ef6a58a0b558d70bd22" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:49:59.486515 907 raft_consensus.cc:1261] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Refusing update from remote peer 72263d79dc214f91b6d58264f83e005a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:49:59.488234 1412 consensus_queue.cc:1035] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fa62c96fa4914471ba169dc0d0dafa9a" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 42229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:49:59.546698 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:49:59.553267 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:49:59.553706 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:00.554839 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:00.561252 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:00.561784 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:01.562716 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:01.568389 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:01.569023 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:02.570003 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:02.578140 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:02.578804 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:03.579840 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:03.585754 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:03.586267 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:04.587247 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:04.592489 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:04.593025 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:05.594053 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:05.599197 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:05.599642 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:06.600688 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:06.606599 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:06.607160 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:07.608593 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:07.617208 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:07.617720 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:08.648829 732 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260806 03:50:08.649573 732 tablet_server.cc:178] TabletServer@127.0.183.1:0 shutting down...
I20260806 03:50:08.690577 732 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:50:08.691437 732 tablet_replica.cc:330] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
I20260806 03:50:08.692490 732 raft_consensus.cc:2229] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.693352 732 raft_consensus.cc:2258] T 0b032f67ab214058a66f3628a92b7b4c P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.696772 732 tablet_replica.cc:330] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
I20260806 03:50:08.697374 732 raft_consensus.cc:2229] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:50:08.698377 732 raft_consensus.cc:2258] T 1219cc03c0af4a0cad5987382c16a506 P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.700379 732 tablet_replica.cc:330] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
I20260806 03:50:08.700943 732 raft_consensus.cc:2229] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.701416 732 raft_consensus.cc:2258] T e02c77493ffe4acbb551c14b3d7a623e P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.703922 732 tablet_replica.cc:330] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
W20260806 03:50:08.704218 955 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260806 03:50:08.704761 732 raft_consensus.cc:2229] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.705473 732 raft_consensus.cc:2258] T 0d84e6173c3c4d85a3e176cd5f33134b P fa62c96fa4914471ba169dc0d0dafa9a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.708194 732 tablet_replica.cc:330] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
W20260806 03:50:08.708084 955 consensus_peers.cc:487] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 -> Peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Couldn't send request to peer fa62c96fa4914471ba169dc0d0dafa9a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:08.708899 732 raft_consensus.cc:2229] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:50:08.709822 732 raft_consensus.cc:2258] T 3d4bbe791ec042679f2862d80343161c P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.711719 732 tablet_replica.cc:330] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
I20260806 03:50:08.712312 732 raft_consensus.cc:2229] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.712814 732 raft_consensus.cc:2258] T 3284b43248b64e15954ecbf158399dea P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.714576 732 tablet_replica.cc:330] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
I20260806 03:50:08.715092 732 raft_consensus.cc:2229] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.715660 732 raft_consensus.cc:2258] T 83229f740c4c49389f6a896927da335c P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.717854 732 tablet_replica.cc:330] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
I20260806 03:50:08.718379 732 raft_consensus.cc:2229] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:50:08.719393 732 raft_consensus.cc:2258] T 58f7c3e941924310a11c59889b06f270 P fa62c96fa4914471ba169dc0d0dafa9a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.721565 732 tablet_replica.cc:330] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a: stopping tablet replica
W20260806 03:50:08.721669 1057 consensus_peers.cc:487] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a -> Peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Couldn't send request to peer fa62c96fa4914471ba169dc0d0dafa9a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:08.722102 732 raft_consensus.cc:2229] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.722625 732 raft_consensus.cc:2258] T c0ffe10b5eb94e78bb6e646f9a84a55d P fa62c96fa4914471ba169dc0d0dafa9a [term 3 FOLLOWER]: Raft consensus is shut down!
W20260806 03:50:08.846454 955 consensus_peers.cc:487] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 -> Peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Couldn't send request to peer fa62c96fa4914471ba169dc0d0dafa9a. Status: Network error: Client connection negotiation failed: client connection to 127.0.183.1:42229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:08.847297 732 tablet_server.cc:194] TabletServer@127.0.183.1:0 shutdown complete.
I20260806 03:50:08.870000 732 tablet_server.cc:178] TabletServer@127.0.183.2:0 shutting down...
W20260806 03:50:08.874854 1057 consensus_peers.cc:487] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a -> Peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Couldn't send request to peer 24b4fc42da314ef6a58a0b558d70bd22. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:08.910271 732 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:50:08.911120 732 tablet_replica.cc:330] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.911809 732 raft_consensus.cc:2229] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.912426 732 raft_consensus.cc:2258] T 58f7c3e941924310a11c59889b06f270 P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.914443 732 tablet_replica.cc:330] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.914995 732 raft_consensus.cc:2229] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.915423 732 raft_consensus.cc:2258] T 1219cc03c0af4a0cad5987382c16a506 P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.917387 732 tablet_replica.cc:330] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.917862 732 raft_consensus.cc:2229] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:50:08.918741 732 raft_consensus.cc:2258] T c0ffe10b5eb94e78bb6e646f9a84a55d P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.920663 732 tablet_replica.cc:330] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.921133 732 raft_consensus.cc:2229] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.921543 732 raft_consensus.cc:2258] T 0d84e6173c3c4d85a3e176cd5f33134b P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.923133 732 tablet_replica.cc:330] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.923588 732 raft_consensus.cc:2229] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.923954 732 raft_consensus.cc:2258] T 3d4bbe791ec042679f2862d80343161c P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.925685 732 tablet_replica.cc:330] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.926126 732 raft_consensus.cc:2229] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.926508 732 raft_consensus.cc:2258] T 83229f740c4c49389f6a896927da335c P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.928269 732 tablet_replica.cc:330] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.928741 732 raft_consensus.cc:2229] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:08.929153 732 raft_consensus.cc:2258] T 3284b43248b64e15954ecbf158399dea P 24b4fc42da314ef6a58a0b558d70bd22 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.931020 732 tablet_replica.cc:330] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.931443 732 raft_consensus.cc:2229] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:50:08.932343 732 raft_consensus.cc:2258] T 0b032f67ab214058a66f3628a92b7b4c P 24b4fc42da314ef6a58a0b558d70bd22 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:08.934149 732 tablet_replica.cc:330] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22: stopping tablet replica
I20260806 03:50:08.934767 732 raft_consensus.cc:2229] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:50:08.935575 732 raft_consensus.cc:2258] T e02c77493ffe4acbb551c14b3d7a623e P 24b4fc42da314ef6a58a0b558d70bd22 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260806 03:50:08.942567 1057 consensus_peers.cc:487] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a -> Peer 24b4fc42da314ef6a58a0b558d70bd22 (127.0.183.2:34259): Couldn't send request to peer 24b4fc42da314ef6a58a0b558d70bd22. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:50:08.943128 1057 consensus_peers.cc:487] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a -> Peer fa62c96fa4914471ba169dc0d0dafa9a (127.0.183.1:42229): Couldn't send request to peer fa62c96fa4914471ba169dc0d0dafa9a. Status: Network error: Client connection negotiation failed: client connection to 127.0.183.1:42229: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:09.014998 732 tablet_server.cc:194] TabletServer@127.0.183.2:0 shutdown complete.
I20260806 03:50:09.055621 732 tablet_server.cc:178] TabletServer@127.0.183.3:0 shutting down...
I20260806 03:50:09.104912 732 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:50:09.106001 732 tablet_replica.cc:330] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.107216 732 raft_consensus.cc:2229] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:09.108193 732 raft_consensus.cc:2258] T c0ffe10b5eb94e78bb6e646f9a84a55d P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.111582 732 tablet_replica.cc:330] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.112510 732 raft_consensus.cc:2229] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:50:09.113974 732 raft_consensus.cc:2258] T 0d84e6173c3c4d85a3e176cd5f33134b P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.117146 732 tablet_replica.cc:330] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.118083 732 raft_consensus.cc:2229] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:50:09.119316 732 raft_consensus.cc:2258] T 83229f740c4c49389f6a896927da335c P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.122174 732 tablet_replica.cc:330] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.123075 732 raft_consensus.cc:2229] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:09.123737 732 raft_consensus.cc:2258] T 3d4bbe791ec042679f2862d80343161c P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.126837 732 tablet_replica.cc:330] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.127791 732 raft_consensus.cc:2229] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:50:09.129171 732 raft_consensus.cc:2258] T 3284b43248b64e15954ecbf158399dea P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.132052 732 tablet_replica.cc:330] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.132902 732 raft_consensus.cc:2229] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:09.133793 732 raft_consensus.cc:2258] T 58f7c3e941924310a11c59889b06f270 P 72263d79dc214f91b6d58264f83e005a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.136533 732 tablet_replica.cc:330] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.137174 732 raft_consensus.cc:2229] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:09.137686 732 raft_consensus.cc:2258] T 1219cc03c0af4a0cad5987382c16a506 P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.139472 732 tablet_replica.cc:330] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.139948 732 raft_consensus.cc:2229] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:09.140511 732 raft_consensus.cc:2258] T 0b032f67ab214058a66f3628a92b7b4c P 72263d79dc214f91b6d58264f83e005a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.142175 732 tablet_replica.cc:330] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a: stopping tablet replica
I20260806 03:50:09.142625 732 raft_consensus.cc:2229] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:09.143039 732 raft_consensus.cc:2258] T e02c77493ffe4acbb551c14b3d7a623e P 72263d79dc214f91b6d58264f83e005a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.164323 732 tablet_server.cc:194] TabletServer@127.0.183.3:0 shutdown complete.
I20260806 03:50:09.187249 732 master.cc:537] Master@127.0.183.62:46101 shutting down...
I20260806 03:50:09.298353 841 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Closing
I20260806 03:50:09.542796 732 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:50:09.543628 732 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:09.544112 732 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04a540520ec3416fbf44eb025f39c0c1: stopping tablet replica
I20260806 03:50:09.565456 732 master.cc:552] Master@127.0.183.62:46101 shutdown complete.
24b4fc42da314ef6a58a0b558d70bd22 4
72263d79dc214f91b6d58264f83e005a 1
fa62c96fa4914471ba169dc0d0dafa9a 4
24b4fc42da314ef6a58a0b558d70bd22 3
72263d79dc214f91b6d58264f83e005a 3
fa62c96fa4914471ba169dc0d0dafa9a 3
24b4fc42da314ef6a58a0b558d70bd22 8
72263d79dc214f91b6d58264f83e005a 1
fa62c96fa4914471ba169dc0d0dafa9a 0
24b4fc42da314ef6a58a0b558d70bd22 3
72263d79dc214f91b6d58264f83e005a 3
fa62c96fa4914471ba169dc0d0dafa9a 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (32246 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260806 03:50:09.615337 732 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.183.62:38901
I20260806 03:50:09.616634 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:50:09.623538 1457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:09.623898 1459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:09.623521 1456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:50:09.625468 732 server_base.cc:1034] running on GCE node
I20260806 03:50:09.626856 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:50:09.627091 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:50:09.627255 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988209627238 us; error 0 us; skew 500 ppm
I20260806 03:50:09.627828 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:50:09.634109 732 webserver.cc:460] Webserver started at http://127.0.183.62:44179/ using document root <none> and password file <none>
I20260806 03:50:09.634810 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:50:09.635061 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:50:09.635408 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:50:09.637498 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/master-0-root/instance:
uuid: "61c6e08dac694b38b8dc3842649f3893"
format_stamp: "Formatted at 2026-08-06 03:50:09 on dist-test-slave-kvfs"
I20260806 03:50:09.642817 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260806 03:50:09.646739 1464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:09.647758 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 03:50:09.648114 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/master-0-root
uuid: "61c6e08dac694b38b8dc3842649f3893"
format_stamp: "Formatted at 2026-08-06 03:50:09 on dist-test-slave-kvfs"
I20260806 03:50:09.648422 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:50:09.668220 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:50:09.669776 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:50:09.718418 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.62:38901
I20260806 03:50:09.718510 1538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.62:38901 every 8 connection(s)
I20260806 03:50:09.723084 1540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:09.734229 1540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893: Bootstrap starting.
I20260806 03:50:09.739820 1540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:09.745066 1540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893: No bootstrap required, opened a new log
I20260806 03:50:09.747489 1540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e08dac694b38b8dc3842649f3893" member_type: VOTER }
I20260806 03:50:09.748152 1540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:09.748411 1540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c6e08dac694b38b8dc3842649f3893, State: Initialized, Role: FOLLOWER
I20260806 03:50:09.749092 1540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e08dac694b38b8dc3842649f3893" member_type: VOTER }
I20260806 03:50:09.749687 1540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:50:09.749943 1540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:50:09.750320 1540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:09.755134 1540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e08dac694b38b8dc3842649f3893" member_type: VOTER }
I20260806 03:50:09.755801 1540 leader_election.cc:302] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61c6e08dac694b38b8dc3842649f3893; no voters:
I20260806 03:50:09.757150 1540 leader_election.cc:288] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:50:09.757548 1544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:09.759145 1544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 1 LEADER]: Becoming Leader. State: Replica: 61c6e08dac694b38b8dc3842649f3893, State: Running, Role: LEADER
I20260806 03:50:09.759907 1544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e08dac694b38b8dc3842649f3893" member_type: VOTER }
I20260806 03:50:09.760562 1540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:50:09.765302 1546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61c6e08dac694b38b8dc3842649f3893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e08dac694b38b8dc3842649f3893" member_type: VOTER } }
I20260806 03:50:09.765304 1548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61c6e08dac694b38b8dc3842649f3893. Latest consensus state: current_term: 1 leader_uuid: "61c6e08dac694b38b8dc3842649f3893" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61c6e08dac694b38b8dc3842649f3893" member_type: VOTER } }
I20260806 03:50:09.766230 1548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [sys.catalog]: This master's current role is: LEADER
I20260806 03:50:09.766773 1546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [sys.catalog]: This master's current role is: LEADER
I20260806 03:50:09.770102 1559 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:50:09.776474 1559 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:50:09.780529 732 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 03:50:09.787519 1559 catalog_manager.cc:1347] Generated new cluster ID: 6cb04731118e462cb9b9e5242a76b591
I20260806 03:50:09.787917 1559 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:50:09.805964 1559 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:50:09.807792 1559 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:50:09.820180 1559 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893: Generated new TSK 0
I20260806 03:50:09.820942 1559 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:50:09.851487 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:50:09.858840 1574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:09.859870 1575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:09.862582 1577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:50:09.862793 732 server_base.cc:1034] running on GCE node
I20260806 03:50:09.864213 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:50:09.864418 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:50:09.864539 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988209864530 us; error 0 us; skew 500 ppm
I20260806 03:50:09.865033 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:50:09.867689 732 webserver.cc:460] Webserver started at http://127.0.183.1:35471/ using document root <none> and password file <none>
I20260806 03:50:09.868295 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:50:09.868477 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:50:09.868708 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:50:09.870057 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-0-root/instance:
uuid: "9edeaaf6653d49eab23847908f835558"
format_stamp: "Formatted at 2026-08-06 03:50:09 on dist-test-slave-kvfs"
I20260806 03:50:09.875267 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260806 03:50:09.879073 1587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:09.879943 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:50:09.880373 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-0-root
uuid: "9edeaaf6653d49eab23847908f835558"
format_stamp: "Formatted at 2026-08-06 03:50:09 on dist-test-slave-kvfs"
I20260806 03:50:09.880688 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:50:09.891179 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:50:09.892781 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:50:09.894454 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:50:09.897071 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:50:09.897253 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:09.897562 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:50:09.897737 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:09.946743 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.1:39021
I20260806 03:50:09.946848 1669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.1:39021 every 8 connection(s)
I20260806 03:50:09.952250 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:50:09.960167 1675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:09.962410 1676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:09.963935 1678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:50:09.964643 732 server_base.cc:1034] running on GCE node
I20260806 03:50:09.966077 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:50:09.966274 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:50:09.966467 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988209966454 us; error 0 us; skew 500 ppm
I20260806 03:50:09.967226 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:50:09.969575 1670 heartbeater.cc:347] Connected to a master server at 127.0.183.62:38901
I20260806 03:50:09.969955 1670 heartbeater.cc:464] Registering TS with master...
I20260806 03:50:09.970561 732 webserver.cc:460] Webserver started at http://127.0.183.2:40433/ using document root <none> and password file <none>
I20260806 03:50:09.970836 1670 heartbeater.cc:511] Master 127.0.183.62:38901 requested a full tablet report, sending...
I20260806 03:50:09.971160 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:50:09.971419 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:50:09.971737 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:50:09.973508 1491 ts_manager.cc:194] Registered new tserver with Master: 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021)
I20260806 03:50:09.973598 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-1-root/instance:
uuid: "a3dcda65a7a141619bcc371f9a6d157e"
format_stamp: "Formatted at 2026-08-06 03:50:09 on dist-test-slave-kvfs"
I20260806 03:50:09.976728 1491 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45666
I20260806 03:50:09.980634 732 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 03:50:09.984695 1686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:09.985520 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 03:50:09.985838 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-1-root
uuid: "a3dcda65a7a141619bcc371f9a6d157e"
format_stamp: "Formatted at 2026-08-06 03:50:09 on dist-test-slave-kvfs"
I20260806 03:50:09.986171 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:50:10.015944 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:50:10.017644 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:50:10.019290 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:50:10.021888 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:50:10.022130 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:10.022388 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:50:10.022559 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:10.072438 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.2:34899
I20260806 03:50:10.072535 1777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.2:34899 every 8 connection(s)
I20260806 03:50:10.077664 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:50:10.086077 1782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:10.088177 1781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:10.091359 1784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:50:10.092451 732 server_base.cc:1034] running on GCE node
I20260806 03:50:10.093562 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:50:10.093791 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:50:10.093989 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988210093970 us; error 0 us; skew 500 ppm
I20260806 03:50:10.094808 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:50:10.095904 1778 heartbeater.cc:347] Connected to a master server at 127.0.183.62:38901
I20260806 03:50:10.096583 1778 heartbeater.cc:464] Registering TS with master...
I20260806 03:50:10.097427 1778 heartbeater.cc:511] Master 127.0.183.62:38901 requested a full tablet report, sending...
I20260806 03:50:10.098507 732 webserver.cc:460] Webserver started at http://127.0.183.3:42451/ using document root <none> and password file <none>
I20260806 03:50:10.099215 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:50:10.099435 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:50:10.099758 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:50:10.099813 1491 ts_manager.cc:194] Registered new tserver with Master: a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:10.101166 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-2-root/instance:
uuid: "2e2cb0a12b264e4190d87895271d3e8d"
format_stamp: "Formatted at 2026-08-06 03:50:10 on dist-test-slave-kvfs"
I20260806 03:50:10.101751 1491 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45674
I20260806 03:50:10.106659 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 03:50:10.110739 1790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:10.111655 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 03:50:10.112011 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-2-root
uuid: "2e2cb0a12b264e4190d87895271d3e8d"
format_stamp: "Formatted at 2026-08-06 03:50:10 on dist-test-slave-kvfs"
I20260806 03:50:10.112392 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785988177119491-732-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:50:10.129119 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:50:10.130650 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:50:10.132351 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:50:10.135255 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:50:10.135449 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:10.135782 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:50:10.135946 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:10.188571 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.3:37601
I20260806 03:50:10.188694 1875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.3:37601 every 8 connection(s)
I20260806 03:50:10.206633 1877 heartbeater.cc:347] Connected to a master server at 127.0.183.62:38901
I20260806 03:50:10.207124 1877 heartbeater.cc:464] Registering TS with master...
I20260806 03:50:10.207957 1877 heartbeater.cc:511] Master 127.0.183.62:38901 requested a full tablet report, sending...
I20260806 03:50:10.210165 1491 ts_manager.cc:194] Registered new tserver with Master: 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:10.210963 732 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018635919s
I20260806 03:50:10.211936 1491 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45688
I20260806 03:50:10.213934 732 test_util.cc:268] Using random seed: -925383663
I20260806 03:50:10.243948 1491 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45702:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:50:10.247434 1491 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:50:10.329538 1624 tablet_service.cc:1467] Processing CreateTablet for tablet 44b9197d3a3e4263b82a245d679b8361 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:50:10.331287 1624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b9197d3a3e4263b82a245d679b8361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.331833 1623 tablet_service.cc:1467] Processing CreateTablet for tablet 507a2c08f1454493ac9114c065b91f1c (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:50:10.333297 1623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507a2c08f1454493ac9114c065b91f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.335074 1616 tablet_service.cc:1467] Processing CreateTablet for tablet dff61d968bfe463f8c94289437b27ba6 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:50:10.336584 1616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff61d968bfe463f8c94289437b27ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.337231 1622 tablet_service.cc:1467] Processing CreateTablet for tablet 9c957f186dbc4ed3892239fac2ca12e4 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:50:10.338714 1622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c957f186dbc4ed3892239fac2ca12e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.341353 1621 tablet_service.cc:1467] Processing CreateTablet for tablet 225db8a5750f4a9d8eead9c679d013a8 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:50:10.342971 1621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225db8a5750f4a9d8eead9c679d013a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.345500 1620 tablet_service.cc:1467] Processing CreateTablet for tablet e940099776f8417eaedcd9339857db23 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:50:10.347239 1620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e940099776f8417eaedcd9339857db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.349639 1617 tablet_service.cc:1467] Processing CreateTablet for tablet 8aae61aedb9448fd8efdb4ea28980244 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:50:10.351357 1617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aae61aedb9448fd8efdb4ea28980244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.354362 1615 tablet_service.cc:1467] Processing CreateTablet for tablet f074e5cca98e4a4ea6eb6bfdc5185353 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:50:10.358106 1612 tablet_service.cc:1467] Processing CreateTablet for tablet 09de95e25a5b47bbb0e59c671908c7ab (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:50:10.359836 1615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f074e5cca98e4a4ea6eb6bfdc5185353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.361501 1612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09de95e25a5b47bbb0e59c671908c7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.376631 1611 tablet_service.cc:1467] Processing CreateTablet for tablet 46f581f68b7a4e6187491b09f936f342 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:50:10.378281 1611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f581f68b7a4e6187491b09f936f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.378393 1825 tablet_service.cc:1467] Processing CreateTablet for tablet 507a2c08f1454493ac9114c065b91f1c (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:50:10.381783 1825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507a2c08f1454493ac9114c065b91f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.381510 1716 tablet_service.cc:1467] Processing CreateTablet for tablet e940099776f8417eaedcd9339857db23 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:50:10.383296 1716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e940099776f8417eaedcd9339857db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.383570 1723 tablet_service.cc:1467] Processing CreateTablet for tablet 507a2c08f1454493ac9114c065b91f1c (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260806 03:50:10.385026 1723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507a2c08f1454493ac9114c065b91f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.385923 1713 tablet_service.cc:1467] Processing CreateTablet for tablet f074e5cca98e4a4ea6eb6bfdc5185353 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:50:10.387430 1713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f074e5cca98e4a4ea6eb6bfdc5185353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.387450 1821 tablet_service.cc:1467] Processing CreateTablet for tablet 8aae61aedb9448fd8efdb4ea28980244 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:50:10.389228 1821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aae61aedb9448fd8efdb4ea28980244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.389547 1823 tablet_service.cc:1467] Processing CreateTablet for tablet 225db8a5750f4a9d8eead9c679d013a8 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:50:10.391045 1823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225db8a5750f4a9d8eead9c679d013a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.390851 1715 tablet_service.cc:1467] Processing CreateTablet for tablet 8aae61aedb9448fd8efdb4ea28980244 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260806 03:50:10.393353 1822 tablet_service.cc:1467] Processing CreateTablet for tablet e940099776f8417eaedcd9339857db23 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260806 03:50:10.394611 1712 tablet_service.cc:1467] Processing CreateTablet for tablet 09de95e25a5b47bbb0e59c671908c7ab (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:50:10.389549 1717 tablet_service.cc:1467] Processing CreateTablet for tablet 225db8a5750f4a9d8eead9c679d013a8 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260806 03:50:10.397267 1711 tablet_service.cc:1467] Processing CreateTablet for tablet 46f581f68b7a4e6187491b09f936f342 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:50:10.399569 1826 tablet_service.cc:1467] Processing CreateTablet for tablet 44b9197d3a3e4263b82a245d679b8361 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:50:10.401834 1819 tablet_service.cc:1467] Processing CreateTablet for tablet f074e5cca98e4a4ea6eb6bfdc5185353 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260806 03:50:10.381644 1725 tablet_service.cc:1467] Processing CreateTablet for tablet 44b9197d3a3e4263b82a245d679b8361 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260806 03:50:10.405045 1817 tablet_service.cc:1467] Processing CreateTablet for tablet 09de95e25a5b47bbb0e59c671908c7ab (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260806 03:50:10.397315 1824 tablet_service.cc:1467] Processing CreateTablet for tablet 9c957f186dbc4ed3892239fac2ca12e4 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:50:10.406360 1715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aae61aedb9448fd8efdb4ea28980244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.399497 1820 tablet_service.cc:1467] Processing CreateTablet for tablet dff61d968bfe463f8c94289437b27ba6 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:50:10.405076 1721 tablet_service.cc:1467] Processing CreateTablet for tablet 9c957f186dbc4ed3892239fac2ca12e4 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260806 03:50:10.408758 1721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c957f186dbc4ed3892239fac2ca12e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.393355 1714 tablet_service.cc:1467] Processing CreateTablet for tablet dff61d968bfe463f8c94289437b27ba6 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260806 03:50:10.410458 1714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff61d968bfe463f8c94289437b27ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.411065 1816 tablet_service.cc:1467] Processing CreateTablet for tablet 46f581f68b7a4e6187491b09f936f342 (DEFAULT_TABLE table=test-workload [id=ddf397f6c5114e318230fbe587f07b1f]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260806 03:50:10.412530 1717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 225db8a5750f4a9d8eead9c679d013a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.414466 1712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09de95e25a5b47bbb0e59c671908c7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.417277 1711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f581f68b7a4e6187491b09f936f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.420452 1725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b9197d3a3e4263b82a245d679b8361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.425205 1822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e940099776f8417eaedcd9339857db23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.428655 1826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44b9197d3a3e4263b82a245d679b8361. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.432626 1820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dff61d968bfe463f8c94289437b27ba6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.436487 1816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46f581f68b7a4e6187491b09f936f342. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.440477 1824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c957f186dbc4ed3892239fac2ca12e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.444350 1819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f074e5cca98e4a4ea6eb6bfdc5185353. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.448446 1817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09de95e25a5b47bbb0e59c671908c7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:10.464062 1898 tablet_bootstrap.cc:492] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:10.491567 1894 tablet_bootstrap.cc:492] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
I20260806 03:50:10.560657 1896 tablet_bootstrap.cc:492] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:10.565266 1894 tablet_bootstrap.cc:654] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.578491 1896 tablet_bootstrap.cc:654] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.628603 1894 tablet_bootstrap.cc:492] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:10.629163 1894 ts_tablet_manager.cc:1397] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.138s user 0.024s sys 0.023s
I20260806 03:50:10.632128 1894 raft_consensus.cc:348] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.632897 1894 raft_consensus.cc:374] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.633185 1894 raft_consensus.cc:729] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.635195 1894 consensus_queue.cc:260] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.655473 1898 tablet_bootstrap.cc:654] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.674908 1896 tablet_bootstrap.cc:492] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:10.675506 1896 ts_tablet_manager.cc:1397] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.115s user 0.017s sys 0.015s
I20260806 03:50:10.679404 1894 ts_tablet_manager.cc:1428] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.050s user 0.016s sys 0.025s
I20260806 03:50:10.679688 1896 raft_consensus.cc:348] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.680789 1778 heartbeater.cc:503] Master 127.0.183.62:38901 was elected leader, sending a full tablet report...
I20260806 03:50:10.680563 1896 raft_consensus.cc:374] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.689101 1896 raft_consensus.cc:729] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.689893 1896 consensus_queue.cc:260] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.693045 1894 tablet_bootstrap.cc:492] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
I20260806 03:50:10.705034 1896 ts_tablet_manager.cc:1428] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.029s user 0.006s sys 0.002s
I20260806 03:50:10.706152 1670 heartbeater.cc:503] Master 127.0.183.62:38901 was elected leader, sending a full tablet report...
I20260806 03:50:10.706423 1896 tablet_bootstrap.cc:492] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:10.709838 1894 tablet_bootstrap.cc:654] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.711404 1898 tablet_bootstrap.cc:492] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:10.711987 1898 ts_tablet_manager.cc:1397] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.249s user 0.027s sys 0.028s
I20260806 03:50:10.715111 1896 tablet_bootstrap.cc:654] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.715363 1898 raft_consensus.cc:348] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.716387 1898 raft_consensus.cc:374] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.716745 1898 raft_consensus.cc:729] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.718070 1902 raft_consensus.cc:482] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:10.717761 1898 consensus_queue.cc:260] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.718734 1902 raft_consensus.cc:504] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.727660 1877 heartbeater.cc:503] Master 127.0.183.62:38901 was elected leader, sending a full tablet report...
I20260806 03:50:10.728106 1898 ts_tablet_manager.cc:1428] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260806 03:50:10.729084 1898 tablet_bootstrap.cc:492] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:10.730525 1896 tablet_bootstrap.cc:492] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:10.731161 1896 ts_tablet_manager.cc:1397] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.006s
I20260806 03:50:10.734387 1896 raft_consensus.cc:348] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.735355 1896 raft_consensus.cc:374] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.735736 1896 raft_consensus.cc:729] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.736860 1896 consensus_queue.cc:260] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.737888 1898 tablet_bootstrap.cc:654] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.739100 1902 leader_election.cc:288] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:10.740212 1896 ts_tablet_manager.cc:1428] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s
I20260806 03:50:10.741653 1896 tablet_bootstrap.cc:492] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:10.750226 1896 tablet_bootstrap.cc:654] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.755539 1898 tablet_bootstrap.cc:492] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:10.757282 1898 ts_tablet_manager.cc:1397] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.013s
I20260806 03:50:10.763101 1894 tablet_bootstrap.cc:492] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:10.763777 1894 ts_tablet_manager.cc:1397] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.071s user 0.036s sys 0.015s
I20260806 03:50:10.764122 1898 raft_consensus.cc:348] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.765125 1898 raft_consensus.cc:374] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.765517 1898 raft_consensus.cc:729] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.766530 1898 consensus_queue.cc:260] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.766855 1894 raft_consensus.cc:348] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.767795 1894 raft_consensus.cc:374] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.768174 1894 raft_consensus.cc:729] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.769003 1894 consensus_queue.cc:260] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.769488 1898 ts_tablet_manager.cc:1428] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.011s user 0.001s sys 0.005s
I20260806 03:50:10.770757 1898 tablet_bootstrap.cc:492] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:10.771320 1894 ts_tablet_manager.cc:1428] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 03:50:10.773595 1894 tablet_bootstrap.cc:492] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
W20260806 03:50:10.774039 1562 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:10.778021 1898 tablet_bootstrap.cc:654] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.779881 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:10.781251 1894 tablet_bootstrap.cc:654] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
W20260806 03:50:10.782039 1687 leader_election.cc:341] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:10.763177 1896 tablet_bootstrap.cc:492] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:10.788130 1896 ts_tablet_manager.cc:1397] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.008s
I20260806 03:50:10.790614 1898 tablet_bootstrap.cc:492] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:10.791183 1898 ts_tablet_manager.cc:1397] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.009s
I20260806 03:50:10.791369 1896 raft_consensus.cc:348] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.792253 1896 raft_consensus.cc:374] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.792591 1896 raft_consensus.cc:729] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.794536 1898 raft_consensus.cc:348] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.796243 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:10.797892 1898 raft_consensus.cc:374] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.798290 1898 raft_consensus.cc:729] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
W20260806 03:50:10.798570 1694 leader_election.cc:341] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:10.799090 1694 leader_election.cc:302] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3dcda65a7a141619bcc371f9a6d157e; no voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558
I20260806 03:50:10.799319 1898 consensus_queue.cc:260] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.797672 1896 consensus_queue.cc:260] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.800725 1902 raft_consensus.cc:2726] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:10.802153 1896 ts_tablet_manager.cc:1428] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260806 03:50:10.802503 1894 tablet_bootstrap.cc:492] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:10.802647 1898 ts_tablet_manager.cc:1428] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.011s user 0.004s sys 0.004s
I20260806 03:50:10.802896 1894 ts_tablet_manager.cc:1397] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.008s
I20260806 03:50:10.803227 1896 tablet_bootstrap.cc:492] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:10.803947 1898 tablet_bootstrap.cc:492] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:10.805625 1894 raft_consensus.cc:348] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.806391 1894 raft_consensus.cc:374] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.806689 1894 raft_consensus.cc:729] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.807466 1894 consensus_queue.cc:260] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.811456 1898 tablet_bootstrap.cc:654] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.812873 1894 ts_tablet_manager.cc:1428] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260806 03:50:10.813982 1894 tablet_bootstrap.cc:492] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
I20260806 03:50:10.816459 1896 tablet_bootstrap.cc:654] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
W20260806 03:50:10.827313 1779 tablet.cc:2369] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:50:10.828678 1894 tablet_bootstrap.cc:654] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.830580 1896 tablet_bootstrap.cc:492] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:10.831151 1896 ts_tablet_manager.cc:1397] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.004s
I20260806 03:50:10.832630 1898 tablet_bootstrap.cc:492] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:10.833343 1898 ts_tablet_manager.cc:1397] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.011s
I20260806 03:50:10.834352 1896 raft_consensus.cc:348] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.835208 1896 raft_consensus.cc:374] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.835521 1896 raft_consensus.cc:729] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.836326 1896 consensus_queue.cc:260] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.836651 1898 raft_consensus.cc:348] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.837422 1898 raft_consensus.cc:374] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.837558 1894 tablet_bootstrap.cc:492] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:10.837729 1898 raft_consensus.cc:729] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.838069 1894 ts_tablet_manager.cc:1397] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.004s
I20260806 03:50:10.838510 1896 ts_tablet_manager.cc:1428] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 03:50:10.838476 1898 consensus_queue.cc:260] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.841149 1898 ts_tablet_manager.cc:1428] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260806 03:50:10.841137 1894 raft_consensus.cc:348] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.842013 1894 raft_consensus.cc:374] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.842258 1896 tablet_bootstrap.cc:492] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:10.842300 1894 raft_consensus.cc:729] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.843076 1894 consensus_queue.cc:260] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.842263 1898 tablet_bootstrap.cc:492] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:10.847770 1894 ts_tablet_manager.cc:1428] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.009s user 0.004s sys 0.004s
I20260806 03:50:10.848927 1902 raft_consensus.cc:482] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:10.849314 1894 tablet_bootstrap.cc:492] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
I20260806 03:50:10.849505 1902 raft_consensus.cc:504] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.850580 1896 tablet_bootstrap.cc:654] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.851046 1898 tablet_bootstrap.cc:654] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.853571 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940099776f8417eaedcd9339857db23" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:10.854513 1842 raft_consensus.cc:2454] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 0.
I20260806 03:50:10.856300 1694 leader_election.cc:302] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:10.858045 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940099776f8417eaedcd9339857db23" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:10.859005 1637 raft_consensus.cc:2454] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 0.
I20260806 03:50:10.866782 1894 tablet_bootstrap.cc:654] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.869218 1926 raft_consensus.cc:2781] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:10.869810 1926 raft_consensus.cc:482] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:10.870230 1926 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:10.871825 1905 raft_consensus.cc:482] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:10.872514 1905 raft_consensus.cc:504] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.876071 1902 leader_election.cc:288] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:10.906647 1905 leader_election.cc:288] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:10.917249 1896 tablet_bootstrap.cc:492] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:10.918097 1896 ts_tablet_manager.cc:1397] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.076s user 0.037s sys 0.027s
I20260806 03:50:10.918028 1926 raft_consensus.cc:504] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.924499 1896 raft_consensus.cc:348] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.925364 1896 raft_consensus.cc:374] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.925662 1896 raft_consensus.cc:729] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.925833 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940099776f8417eaedcd9339857db23" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:10.926479 1637 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:10.927582 1894 tablet_bootstrap.cc:492] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:10.928341 1896 consensus_queue.cc:260] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.930775 1896 ts_tablet_manager.cc:1428] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.012s user 0.005s sys 0.003s
I20260806 03:50:10.931830 1896 tablet_bootstrap.cc:492] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:10.936180 1894 ts_tablet_manager.cc:1397] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.087s user 0.018s sys 0.005s
I20260806 03:50:10.938473 1898 tablet_bootstrap.cc:492] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:10.939126 1898 ts_tablet_manager.cc:1397] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.097s user 0.020s sys 0.039s
I20260806 03:50:10.939488 1894 raft_consensus.cc:348] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.940349 1894 raft_consensus.cc:374] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.940653 1894 raft_consensus.cc:729] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.941524 1894 consensus_queue.cc:260] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.943008 1926 leader_election.cc:288] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:10.936299 1637 raft_consensus.cc:2454] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 1.
I20260806 03:50:10.946259 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940099776f8417eaedcd9339857db23" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:10.946894 1896 tablet_bootstrap.cc:654] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.946914 1842 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:10.947104 1687 leader_election.cc:302] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:10.947039 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:10.943986 1894 ts_tablet_manager.cc:1428] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.007s user 0.001s sys 0.004s
I20260806 03:50:10.943858 1898 raft_consensus.cc:348] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.954042 1898 raft_consensus.cc:374] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.954424 1898 raft_consensus.cc:729] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.954571 1842 raft_consensus.cc:2454] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 1.
I20260806 03:50:10.949131 1894 tablet_bootstrap.cc:492] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
W20260806 03:50:10.949374 1590 leader_election.cc:341] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:10.952891 1926 raft_consensus.cc:2781] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:10.956655 1926 raft_consensus.cc:686] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:10.957572 1926 consensus_queue.cc:237] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.960405 1898 consensus_queue.cc:260] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.961560 1894 tablet_bootstrap.cc:654] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.962383 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:10.963197 1842 raft_consensus.cc:2454] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 0.
I20260806 03:50:10.963806 1898 ts_tablet_manager.cc:1428] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.024s user 0.004s sys 0.004s
I20260806 03:50:10.964963 1898 tablet_bootstrap.cc:492] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:10.964679 1592 leader_election.cc:302] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters: a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:10.966389 1905 raft_consensus.cc:2781] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:10.966766 1905 raft_consensus.cc:482] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:10.967089 1905 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:10.980736 1489 catalog_manager.cc:5495] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 1 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:10.981026 1905 raft_consensus.cc:504] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.987040 1898 tablet_bootstrap.cc:654] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:10.988696 1894 tablet_bootstrap.cc:492] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:10.989176 1894 ts_tablet_manager.cc:1397] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.007s
I20260806 03:50:10.995818 1894 raft_consensus.cc:348] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:10.996436 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:10.996505 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:10.996929 1894 raft_consensus.cc:374] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:10.997151 1842 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:10.997501 1894 raft_consensus.cc:729] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:10.998359 1894 consensus_queue.cc:260] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
W20260806 03:50:10.998768 1590 leader_election.cc:341] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.001052 1894 ts_tablet_manager.cc:1428] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260806 03:50:11.002141 1894 tablet_bootstrap.cc:492] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
I20260806 03:50:11.004515 1898 tablet_bootstrap.cc:492] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:11.005214 1898 ts_tablet_manager.cc:1397] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.040s user 0.009s sys 0.008s
I20260806 03:50:11.005534 1842 raft_consensus.cc:2454] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 1.
I20260806 03:50:11.007012 1592 leader_election.cc:302] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters: a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.008867 1898 raft_consensus.cc:348] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.010195 1898 raft_consensus.cc:374] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.010195 1896 tablet_bootstrap.cc:492] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:11.010598 1898 raft_consensus.cc:729] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.010916 1896 ts_tablet_manager.cc:1397] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.008s
I20260806 03:50:11.011307 1894 tablet_bootstrap.cc:654] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.011866 1905 leader_election.cc:288] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.011469 1898 consensus_queue.cc:260] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.013150 1905 raft_consensus.cc:2781] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.017871 1898 ts_tablet_manager.cc:1428] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260806 03:50:11.020445 1898 tablet_bootstrap.cc:492] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:11.021577 1896 raft_consensus.cc:348] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.022851 1896 raft_consensus.cc:374] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.023334 1896 raft_consensus.cc:729] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.024560 1896 consensus_queue.cc:260] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.019071 1905 raft_consensus.cc:686] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:11.028744 1898 tablet_bootstrap.cc:654] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.040179 1896 ts_tablet_manager.cc:1428] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.029s user 0.010s sys 0.008s
I20260806 03:50:11.049189 1896 tablet_bootstrap.cc:492] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:11.053009 1905 consensus_queue.cc:237] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.050947 1943 raft_consensus.cc:482] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:50:11.056260 1592 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:37601, user_credentials={real_user=slave}} blocked reactor thread for 49298.2us
I20260806 03:50:11.056336 1943 raft_consensus.cc:504] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.062212 1926 raft_consensus.cc:482] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.062822 1926 raft_consensus.cc:504] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.065586 1898 tablet_bootstrap.cc:492] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:11.065778 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507a2c08f1454493ac9114c065b91f1c" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:11.066224 1898 ts_tablet_manager.cc:1397] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.046s user 0.028s sys 0.012s
I20260806 03:50:11.066581 1741 raft_consensus.cc:2454] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 0.
I20260806 03:50:11.066835 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.067229 1940 raft_consensus.cc:482] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.067528 1842 raft_consensus.cc:2454] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 0.
I20260806 03:50:11.067782 1940 raft_consensus.cc:504] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.068400 1590 leader_election.cc:302] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.068885 1694 leader_election.cc:302] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.069731 1926 leader_election.cc:288] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.069410 1898 raft_consensus.cc:348] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.070207 1898 raft_consensus.cc:374] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.070597 1940 leader_election.cc:288] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.070564 1898 raft_consensus.cc:729] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.069758 1902 raft_consensus.cc:2781] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:11.071208 1940 raft_consensus.cc:2781] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:11.071224 1902 raft_consensus.cc:482] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:11.071381 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.071882 1902 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.071677 1898 consensus_queue.cc:260] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
W20260806 03:50:11.072857 1592 leader_election.cc:341] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.071686 1940 raft_consensus.cc:482] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:11.073649 1940 raft_consensus.cc:3037] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.074698 1894 tablet_bootstrap.cc:492] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:11.075204 1894 ts_tablet_manager.cc:1397] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.073s user 0.025s sys 0.029s
I20260806 03:50:11.076592 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:11.077858 1898 ts_tablet_manager.cc:1428] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
W20260806 03:50:11.078184 1590 leader_election.cc:341] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.078248 1894 raft_consensus.cc:348] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.078725 1590 leader_election.cc:302] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9edeaaf6653d49eab23847908f835558; no voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.079173 1898 tablet_bootstrap.cc:492] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:11.079075 1894 raft_consensus.cc:374] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.081921 1894 raft_consensus.cc:729] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.082818 1894 consensus_queue.cc:260] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.085250 1902 raft_consensus.cc:504] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.086745 1898 tablet_bootstrap.cc:654] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.088587 1894 ts_tablet_manager.cc:1428] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.013s user 0.008s sys 0.002s
I20260806 03:50:11.089709 1894 tablet_bootstrap.cc:492] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
I20260806 03:50:11.089859 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:11.091687 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558"
W20260806 03:50:11.093294 1687 leader_election.cc:341] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.096642 1894 tablet_bootstrap.cc:654] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
W20260806 03:50:11.098861 1687 leader_election.cc:341] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.099453 1898 tablet_bootstrap.cc:492] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:11.099992 1898 ts_tablet_manager.cc:1397] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.004s
I20260806 03:50:11.100976 1902 leader_election.cc:288] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.102249 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.101858 1943 leader_election.cc:288] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.103101 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507a2c08f1454493ac9114c065b91f1c" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.107689 1842 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.109249 1943 raft_consensus.cc:482] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.109939 1943 raft_consensus.cc:504] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.103018 1898 raft_consensus.cc:348] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.112214 1898 raft_consensus.cc:374] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.106221 1896 tablet_bootstrap.cc:654] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.112675 1898 raft_consensus.cc:729] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.111866 1491 catalog_manager.cc:5495] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.113379 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:11.113569 1898 consensus_queue.cc:260] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
W20260806 03:50:11.115252 1590 leader_election.cc:341] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.106884 1945 raft_consensus.cc:2726] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:11.115787 1898 ts_tablet_manager.cc:1428] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.015s user 0.002s sys 0.002s
I20260806 03:50:11.122340 1898 tablet_bootstrap.cc:492] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:11.106324 1894 tablet_bootstrap.cc:492] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:11.128530 1894 ts_tablet_manager.cc:1397] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.039s user 0.015s sys 0.000s
I20260806 03:50:11.129540 1940 raft_consensus.cc:504] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.116853 1943 leader_election.cc:288] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.121073 1842 raft_consensus.cc:2454] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 1.
I20260806 03:50:11.131532 1894 raft_consensus.cc:348] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.132282 1894 raft_consensus.cc:374] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.132347 1694 leader_election.cc:302] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters: 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.132660 1894 raft_consensus.cc:729] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.133423 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507a2c08f1454493ac9114c065b91f1c" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:11.133813 1902 raft_consensus.cc:2781] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.134280 1741 raft_consensus.cc:3037] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.135380 1902 raft_consensus.cc:686] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.135725 1898 tablet_bootstrap.cc:654] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.136534 1902 consensus_queue.cc:237] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.148175 1898 tablet_bootstrap.cc:492] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:11.148909 1898 ts_tablet_manager.cc:1397] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.008s
I20260806 03:50:11.149473 1741 raft_consensus.cc:2454] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 1.
I20260806 03:50:11.150867 1590 leader_election.cc:302] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.150867 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.151973 1950 raft_consensus.cc:2781] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.152275 1842 raft_consensus.cc:2454] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 0.
I20260806 03:50:11.152803 1950 raft_consensus.cc:686] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
W20260806 03:50:11.153735 1592 leader_election.cc:341] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.153810 1950 consensus_queue.cc:237] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.154882 1592 leader_election.cc:302] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters: a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.151976 1898 raft_consensus.cc:348] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.155507 1898 raft_consensus.cc:374] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.155706 1943 raft_consensus.cc:2781] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:11.155802 1898 raft_consensus.cc:729] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.156186 1943 raft_consensus.cc:482] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:11.156565 1943 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.156697 1898 consensus_queue.cc:260] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.157727 1940 leader_election.cc:288] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.133313 1894 consensus_queue.cc:260] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.160382 1894 ts_tablet_manager.cc:1428] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260806 03:50:11.161346 1894 tablet_bootstrap.cc:492] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
I20260806 03:50:11.161540 1898 ts_tablet_manager.cc:1428] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260806 03:50:11.162569 1898 tablet_bootstrap.cc:492] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d: Bootstrap starting.
I20260806 03:50:11.164619 1943 raft_consensus.cc:504] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.168311 1898 tablet_bootstrap.cc:654] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.168838 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507a2c08f1454493ac9114c065b91f1c" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.169481 1842 raft_consensus.cc:3037] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.170665 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
W20260806 03:50:11.172772 1590 leader_election.cc:341] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.175736 1894 tablet_bootstrap.cc:654] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.178076 1491 catalog_manager.cc:5495] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 1 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.180922 1842 raft_consensus.cc:2454] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 1.
I20260806 03:50:11.183504 1902 raft_consensus.cc:482] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.184136 1943 leader_election.cc:288] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.184314 1902 raft_consensus.cc:504] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.186244 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.187291 1839 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.189698 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
W20260806 03:50:11.192169 1687 leader_election.cc:341] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.195484 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.196389 1838 raft_consensus.cc:2454] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 0.
I20260806 03:50:11.197826 1694 leader_election.cc:302] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters: 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.198042 1839 raft_consensus.cc:2454] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 1.
I20260806 03:50:11.198836 1926 raft_consensus.cc:2781] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:11.199221 1926 raft_consensus.cc:482] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:11.199569 1926 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.199785 1592 leader_election.cc:302] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters: a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.201654 1943 raft_consensus.cc:2781] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.202256 1943 raft_consensus.cc:686] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:11.203326 1943 consensus_queue.cc:237] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.209550 1896 tablet_bootstrap.cc:492] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:11.210273 1896 ts_tablet_manager.cc:1397] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.162s user 0.020s sys 0.009s
I20260806 03:50:11.210116 1902 leader_election.cc:288] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.222790 1909 raft_consensus.cc:482] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.214732 1896 raft_consensus.cc:348] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.223501 1909 raft_consensus.cc:504] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.224284 1896 raft_consensus.cc:374] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.224684 1896 raft_consensus.cc:729] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.225508 1896 consensus_queue.cc:260] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.225281 1487 catalog_manager.cc:5495] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.227627 1894 tablet_bootstrap.cc:492] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:11.228283 1894 ts_tablet_manager.cc:1397] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.067s user 0.015s sys 0.015s
I20260806 03:50:11.228524 1896 ts_tablet_manager.cc:1428] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.018s user 0.007s sys 0.004s
I20260806 03:50:11.223737 1926 raft_consensus.cc:504] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.229575 1896 tablet_bootstrap.cc:492] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:11.229858 1909 leader_election.cc:288] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:11.237092 1926 leader_election.cc:288] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.237504 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558"
W20260806 03:50:11.239554 1687 leader_election.cc:341] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.239971 1898 tablet_bootstrap.cc:492] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d: No bootstrap required, opened a new log
I20260806 03:50:11.240490 1898 ts_tablet_manager.cc:1397] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent bootstrapping tablet: real 0.078s user 0.019s sys 0.036s
I20260806 03:50:11.243497 1898 raft_consensus.cc:348] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.244529 1898 raft_consensus.cc:374] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.244827 1898 raft_consensus.cc:729] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.245502 1898 consensus_queue.cc:260] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.232507 1894 raft_consensus.cc:348] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.248323 1894 raft_consensus.cc:374] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.248750 1894 raft_consensus.cc:729] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.249439 1894 consensus_queue.cc:260] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.247987 1898 ts_tablet_manager.cc:1428] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260806 03:50:11.253151 1894 ts_tablet_manager.cc:1428] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260806 03:50:11.254293 1894 tablet_bootstrap.cc:492] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e: Bootstrap starting.
I20260806 03:50:11.256484 1909 raft_consensus.cc:482] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.257455 1909 raft_consensus.cc:504] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.258090 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.258693 1839 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.259439 1909 leader_election.cc:288] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:11.257206 1896 tablet_bootstrap.cc:654] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.274461 1491 catalog_manager.cc:5495] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.280687 1894 tablet_bootstrap.cc:654] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.281565 1896 tablet_bootstrap.cc:492] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:11.282418 1896 ts_tablet_manager.cc:1397] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.007s
I20260806 03:50:11.285600 1896 raft_consensus.cc:348] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.286494 1896 raft_consensus.cc:374] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.287180 1896 raft_consensus.cc:729] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.287989 1839 raft_consensus.cc:2454] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 1.
I20260806 03:50:11.288442 1896 consensus_queue.cc:260] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.289140 1694 leader_election.cc:302] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters: 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.290349 1926 raft_consensus.cc:2781] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.290846 1926 raft_consensus.cc:686] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.291426 1896 ts_tablet_manager.cc:1428] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260806 03:50:11.291706 1926 consensus_queue.cc:237] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.292742 1896 tablet_bootstrap.cc:492] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:11.297731 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09de95e25a5b47bbb0e59c671908c7ab" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:11.298849 1740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:11.299314 1741 raft_consensus.cc:2454] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 0.
I20260806 03:50:11.299626 1740 raft_consensus.cc:2454] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 0.
I20260806 03:50:11.302418 1793 leader_election.cc:302] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.303941 1793 leader_election.cc:302] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.305195 1909 raft_consensus.cc:2781] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:11.305656 1909 raft_consensus.cc:482] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:11.306094 1909 raft_consensus.cc:3037] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.305974 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09de95e25a5b47bbb0e59c671908c7ab" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:11.306491 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:11.306780 1637 raft_consensus.cc:2454] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 0.
I20260806 03:50:11.307171 1636 raft_consensus.cc:2454] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 0.
I20260806 03:50:11.313657 1902 raft_consensus.cc:482] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.314222 1902 raft_consensus.cc:504] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.317062 1902 leader_election.cc:288] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.318017 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:11.319067 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.319741 1839 raft_consensus.cc:2454] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 0.
W20260806 03:50:11.319974 1687 leader_election.cc:341] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.321066 1694 leader_election.cc:302] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters: 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.322479 1926 raft_consensus.cc:2781] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:11.323273 1926 raft_consensus.cc:482] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:11.323396 1967 raft_consensus.cc:2781] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:11.323858 1967 raft_consensus.cc:482] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:11.323912 1926 raft_consensus.cc:3037] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.324246 1967 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.329023 1487 catalog_manager.cc:5495] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 1 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.329902 1909 raft_consensus.cc:504] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.333858 1909 leader_election.cc:288] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 1 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:11.333843 1926 raft_consensus.cc:504] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.336203 1926 leader_election.cc:288] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.337038 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:11.337116 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09de95e25a5b47bbb0e59c671908c7ab" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:11.337790 1636 raft_consensus.cc:3037] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.339258 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.339888 1839 raft_consensus.cc:3037] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.336229 1740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09de95e25a5b47bbb0e59c671908c7ab" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:11.344610 1740 raft_consensus.cc:3037] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.344790 1894 tablet_bootstrap.cc:492] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e: No bootstrap required, opened a new log
I20260806 03:50:11.345260 1894 ts_tablet_manager.cc:1397] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e: Time spent bootstrapping tablet: real 0.091s user 0.010s sys 0.019s
W20260806 03:50:11.346067 1687 leader_election.cc:341] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:11.347198 1967 raft_consensus.cc:504] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.350473 1967 leader_election.cc:288] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 1 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:11.350373 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:11.351043 1637 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.351856 1926 consensus_queue.cc:1035] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:11.352164 1740 raft_consensus.cc:2454] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 1.
I20260806 03:50:11.353346 1636 raft_consensus.cc:2454] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 1.
I20260806 03:50:11.353561 1793 leader_election.cc:302] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.355429 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:11.355947 1909 raft_consensus.cc:2781] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.356096 1741 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.356505 1839 raft_consensus.cc:2454] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 1.
I20260806 03:50:11.358083 1694 leader_election.cc:302] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters: 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.359979 1896 tablet_bootstrap.cc:654] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.360430 1926 raft_consensus.cc:2781] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.363726 1926 raft_consensus.cc:686] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.363794 1894 raft_consensus.cc:348] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.364913 1894 raft_consensus.cc:374] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.365411 1894 raft_consensus.cc:729] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.364944 1926 consensus_queue.cc:237] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.366278 1894 consensus_queue.cc:260] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.368110 1637 raft_consensus.cc:2454] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 1.
I20260806 03:50:11.366274 1741 raft_consensus.cc:2454] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 1.
I20260806 03:50:11.369619 1791 leader_election.cc:302] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters:
I20260806 03:50:11.369135 1894 ts_tablet_manager.cc:1428] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e: Time spent starting tablet: real 0.024s user 0.009s sys 0.000s
I20260806 03:50:11.371840 1967 raft_consensus.cc:2781] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.372639 1967 raft_consensus.cc:686] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Becoming Leader. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:11.372727 1902 consensus_queue.cc:1035] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:11.374197 1967 consensus_queue.cc:237] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.375454 1909 raft_consensus.cc:686] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Becoming Leader. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:11.376940 1909 consensus_queue.cc:237] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.418174 1896 tablet_bootstrap.cc:492] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:11.418730 1896 ts_tablet_manager.cc:1397] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.126s user 0.025s sys 0.020s
I20260806 03:50:11.422752 1487 catalog_manager.cc:5495] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3). New cstate: current_term: 1 leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:11.426386 1896 raft_consensus.cc:348] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.430333 1491 catalog_manager.cc:5495] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 1 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.432757 1896 raft_consensus.cc:374] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.433204 1896 raft_consensus.cc:729] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.433930 1896 consensus_queue.cc:260] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.436412 1896 ts_tablet_manager.cc:1428] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.017s user 0.010s sys 0.000s
I20260806 03:50:11.442200 1896 tablet_bootstrap.cc:492] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558: Bootstrap starting.
I20260806 03:50:11.445505 1489 catalog_manager.cc:5495] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3). New cstate: current_term: 1 leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:11.448679 1896 tablet_bootstrap.cc:654] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:11.459981 1896 tablet_bootstrap.cc:492] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558: No bootstrap required, opened a new log
I20260806 03:50:11.460548 1896 ts_tablet_manager.cc:1397] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.009s
I20260806 03:50:11.464130 1896 raft_consensus.cc:348] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.464864 1896 raft_consensus.cc:374] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:11.465134 1896 raft_consensus.cc:729] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Initialized, Role: FOLLOWER
I20260806 03:50:11.465752 1896 consensus_queue.cc:260] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.467813 1896 ts_tablet_manager.cc:1428] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20260806 03:50:11.499349 1902 raft_consensus.cc:482] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.500126 1902 raft_consensus.cc:504] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.502115 1902 leader_election.cc:288] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.503286 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.503157 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:11.504076 1839 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 0.
I20260806 03:50:11.504076 1636 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 0.
I20260806 03:50:11.505396 1694 leader_election.cc:302] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.506163 1902 raft_consensus.cc:2781] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:11.506515 1902 raft_consensus.cc:482] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:11.506809 1902 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.512323 1902 raft_consensus.cc:504] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.514278 1902 leader_election.cc:288] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.515468 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.515468 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:11.516243 1839 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.516327 1636 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.521440 1636 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 1.
I20260806 03:50:11.521479 1839 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 1.
I20260806 03:50:11.521765 1943 raft_consensus.cc:482] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.522696 1687 leader_election.cc:302] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.522315 1943 raft_consensus.cc:504] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.523507 1902 raft_consensus.cc:2781] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:11.524057 1902 raft_consensus.cc:686] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.524721 1943 leader_election.cc:288] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.524924 1902 consensus_queue.cc:237] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.525885 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:11.527258 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.528097 1839 raft_consensus.cc:2379] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9edeaaf6653d49eab23847908f835558 in current term 1: Already voted for candidate a3dcda65a7a141619bcc371f9a6d157e in this term.
I20260806 03:50:11.529163 1592 leader_election.cc:302] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9edeaaf6653d49eab23847908f835558; no voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.529918 1943 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.533586 1491 catalog_manager.cc:5495] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 0 to 1, leader changed from <none> to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 1 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.535349 1943 raft_consensus.cc:2726] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:11.575354 1741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:53318
I20260806 03:50:11.576277 1741 raft_consensus.cc:595] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.578536 1943 consensus_queue.cc:1035] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.584159 1902 raft_consensus.cc:482] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.584728 1902 raft_consensus.cc:504] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.585355 1741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:53318
I20260806 03:50:11.586005 1741 raft_consensus.cc:595] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.586850 1902 leader_election.cc:288] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.589820 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.590549 1741 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.590703 1838 raft_consensus.cc:2379] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a3dcda65a7a141619bcc371f9a6d157e in current term 1: Already voted for candidate 9edeaaf6653d49eab23847908f835558 in this term.
I20260806 03:50:11.593382 1943 consensus_queue.cc:1035] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.597720 1737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:53318
I20260806 03:50:11.598381 1737 raft_consensus.cc:595] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.598729 1950 consensus_queue.cc:1035] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.604466 1950 consensus_queue.cc:1035] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:11.607312 1741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e940099776f8417eaedcd9339857db23"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:53318
I20260806 03:50:11.607986 1741 raft_consensus.cc:595] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.609614 1905 consensus_queue.cc:1035] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.609772 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:11.613639 1687 leader_election.cc:302] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3dcda65a7a141619bcc371f9a6d157e; no voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558
I20260806 03:50:11.614543 1902 raft_consensus.cc:2726] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:11.641004 1636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:40666
I20260806 03:50:11.641988 1636 raft_consensus.cc:595] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Received request to transfer leadership to TS a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.645239 1741 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.652009 1943 raft_consensus.cc:482] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.652673 1943 raft_consensus.cc:504] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.653093 1636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:40666
I20260806 03:50:11.653756 1636 raft_consensus.cc:595] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Received request to transfer leadership to TS a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.653721 1950 consensus_queue.cc:1035] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:11.654718 1943 leader_election.cc:288] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.655771 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:11.657403 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.658272 1839 raft_consensus.cc:2379] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9edeaaf6653d49eab23847908f835558 in current term 1: Already voted for candidate a3dcda65a7a141619bcc371f9a6d157e in this term.
I20260806 03:50:11.659348 1592 leader_election.cc:302] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9edeaaf6653d49eab23847908f835558; no voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.660389 1950 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.665366 1905 raft_consensus.cc:979] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558: : Instructing follower a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:11.665860 1905 raft_consensus.cc:1067] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Signalling peer a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:11.668773 1737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:44480
I20260806 03:50:11.669334 1737 raft_consensus.cc:482] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:11.669730 1737 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.670388 1950 raft_consensus.cc:2726] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:11.675678 1737 raft_consensus.cc:504] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.678774 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:11.679436 1636 raft_consensus.cc:3032] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:11.679780 1636 raft_consensus.cc:729] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:11.680763 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.680692 1636 consensus_queue.cc:260] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.681531 1839 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.682135 1636 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.679798 1737 leader_election.cc:288] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.689743 1839 raft_consensus.cc:2454] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:11.689678 1902 consensus_queue.cc:1035] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:11.690757 1838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
mode: GRACEFUL
new_leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:56540
I20260806 03:50:11.691406 1838 raft_consensus.cc:595] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Received request to transfer leadership to TS a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.691763 1694 leader_election.cc:302] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.692085 1636 raft_consensus.cc:2454] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:11.693367 1902 raft_consensus.cc:2781] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:11.694106 1902 raft_consensus.cc:686] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.695207 1902 consensus_queue.cc:237] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.701187 1979 consensus_queue.cc:1035] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:11.701200 1838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "09de95e25a5b47bbb0e59c671908c7ab"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
mode: GRACEFUL
new_leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:56540
I20260806 03:50:11.702181 1838 raft_consensus.cc:595] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Received request to transfer leadership to TS a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.703562 1979 consensus_queue.cc:1035] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.708495 1491 catalog_manager.cc:5495] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 1 to 2, leader changed from 9edeaaf6653d49eab23847908f835558 (127.0.183.1) to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 2 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.712903 1737 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
mode: GRACEFUL
new_leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:53318
I20260806 03:50:11.713596 1737 raft_consensus.cc:595] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Received request to transfer leadership to TS 2e2cb0a12b264e4190d87895271d3e8d
I20260806 03:50:11.715929 1950 raft_consensus.cc:482] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:11.716421 1950 raft_consensus.cc:504] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.718587 1950 leader_election.cc:288] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.719890 1979 raft_consensus.cc:979] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:11.720149 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:11.720994 1839 raft_consensus.cc:2379] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9edeaaf6653d49eab23847908f835558 in current term 1: Already voted for candidate a3dcda65a7a141619bcc371f9a6d157e in this term.
I20260806 03:50:11.720343 1979 raft_consensus.cc:1067] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:11.721948 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:11.723335 1590 leader_election.cc:302] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9edeaaf6653d49eab23847908f835558; no voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:11.724671 1950 raft_consensus.cc:3037] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:11.727923 1636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43176
I20260806 03:50:11.729803 1636 raft_consensus.cc:482] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:11.730232 1636 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.731921 1902 consensus_queue.cc:1035] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.738125 1636 raft_consensus.cc:504] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.740702 1636 leader_election.cc:288] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 2 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.742791 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:11.742668 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.743516 1737 raft_consensus.cc:3032] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:11.743621 1838 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.743850 1737 raft_consensus.cc:729] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.744891 1737 consensus_queue.cc:260] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.746181 1737 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.752943 1838 raft_consensus.cc:2454] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 2.
I20260806 03:50:11.754218 1592 leader_election.cc:302] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters:
I20260806 03:50:11.754936 1950 raft_consensus.cc:2726] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:11.755404 1943 raft_consensus.cc:2781] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:11.755939 1943 raft_consensus.cc:686] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:11.756912 1943 consensus_queue.cc:237] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.758546 1979 raft_consensus.cc:979] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:11.759014 1926 raft_consensus.cc:1067] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:11.762912 1636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43176
I20260806 03:50:11.763450 1636 raft_consensus.cc:482] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:11.763749 1636 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.770020 1636 raft_consensus.cc:504] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.770069 1491 catalog_manager.cc:5495] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 1 to 2, leader changed from a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 2 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.772328 1967 consensus_queue.cc:1035] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.769924 1737 raft_consensus.cc:2454] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 2.
I20260806 03:50:11.773972 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:11.774622 1741 raft_consensus.cc:3032] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:11.774691 1637 raft_consensus.cc:1226] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Rejecting Update request from peer a3dcda65a7a141619bcc371f9a6d157e for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:11.774979 1741 raft_consensus.cc:729] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.775848 1741 consensus_queue.cc:260] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.778226 1741 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:50:11.780174 1902 consensus_queue.cc:1159] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "9edeaaf6653d49eab23847908f835558" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 0 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer a3dcda65a7a141619bcc371f9a6d157e for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:50:11.788138 1741 raft_consensus.cc:2454] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 2.
I20260806 03:50:11.789004 1636 leader_election.cc:288] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 2 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.790603 1590 leader_election.cc:302] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.791591 1950 raft_consensus.cc:2781] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:11.792148 1950 raft_consensus.cc:686] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:11.793102 1950 consensus_queue.cc:237] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.794981 1909 consensus_queue.cc:1035] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.796360 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.797082 1838 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.805055 1909 raft_consensus.cc:979] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d: : Instructing follower a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:11.806427 1909 raft_consensus.cc:1067] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Signalling peer a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:11.807081 1491 catalog_manager.cc:5495] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 1 to 2, leader changed from a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 2 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.807976 1838 raft_consensus.cc:2454] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 2.
I20260806 03:50:11.808207 1737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:44496
I20260806 03:50:11.808902 1737 raft_consensus.cc:482] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:11.809288 1737 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.816334 1737 raft_consensus.cc:504] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.818580 1737 leader_election.cc:288] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.819703 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.820159 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:11.820420 1838 raft_consensus.cc:3032] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:11.820765 1636 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.820822 1838 raft_consensus.cc:729] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:11.821864 1838 consensus_queue.cc:260] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.823216 1838 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.830994 1636 raft_consensus.cc:2454] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:11.832170 1687 leader_election.cc:302] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.832949 1902 raft_consensus.cc:2781] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:11.833386 1902 raft_consensus.cc:686] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.834285 1838 raft_consensus.cc:2454] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:11.834237 1902 consensus_queue.cc:237] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.841964 1489 catalog_manager.cc:5495] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 1 to 2, leader changed from 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3) to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 2 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.846849 1902 consensus_queue.cc:1035] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.854522 1926 raft_consensus.cc:979] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e: : Instructing follower 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:11.855048 1979 raft_consensus.cc:1067] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Signalling peer 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:11.857841 1839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:38726
I20260806 03:50:11.859189 1839 raft_consensus.cc:482] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:11.859587 1839 raft_consensus.cc:3037] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.863840 1902 consensus_queue.cc:1035] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.869663 1839 raft_consensus.cc:504] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.873870 1839 leader_election.cc:288] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 2 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:11.874779 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:11.875025 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46f581f68b7a4e6187491b09f936f342" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:11.875425 1636 raft_consensus.cc:3037] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.875676 1737 raft_consensus.cc:3032] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:11.876021 1737 raft_consensus.cc:729] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.876724 1737 consensus_queue.cc:260] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.877717 1737 raft_consensus.cc:3037] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.881214 1636 raft_consensus.cc:2454] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 2.
I20260806 03:50:11.883569 1737 raft_consensus.cc:2454] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 2.
I20260806 03:50:11.883828 1791 leader_election.cc:302] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters:
I20260806 03:50:11.884809 1909 raft_consensus.cc:2781] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:11.885320 1909 raft_consensus.cc:686] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 LEADER]: Becoming Leader. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:11.886237 1909 consensus_queue.cc:237] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.892874 1909 consensus_queue.cc:1035] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.895144 1491 catalog_manager.cc:5495] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d reported cstate change: term changed from 1 to 2, leader changed from a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2) to 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3). New cstate: current_term: 2 leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:11.904986 2014 consensus_queue.cc:1035] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:11.914503 2014 raft_consensus.cc:979] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d: : Instructing follower a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:11.915213 2014 raft_consensus.cc:1067] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Signalling peer a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:11.917030 1737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "09de95e25a5b47bbb0e59c671908c7ab"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:44496
I20260806 03:50:11.917582 1737 raft_consensus.cc:482] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:11.917909 1737 raft_consensus.cc:3037] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.923645 1737 raft_consensus.cc:504] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.925904 1737 leader_election.cc:288] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.926640 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09de95e25a5b47bbb0e59c671908c7ab" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.927444 1839 raft_consensus.cc:3032] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:11.926561 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09de95e25a5b47bbb0e59c671908c7ab" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:11.927754 1839 raft_consensus.cc:729] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:11.927943 1636 raft_consensus.cc:3037] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.928653 1839 consensus_queue.cc:260] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.930047 1839 raft_consensus.cc:3037] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.935750 1636 raft_consensus.cc:2454] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:11.937134 1687 leader_election.cc:302] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:11.937325 1839 raft_consensus.cc:2454] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:11.937942 1902 raft_consensus.cc:2781] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:11.938428 1902 raft_consensus.cc:686] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.939324 1902 consensus_queue.cc:237] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.946111 1489 catalog_manager.cc:5495] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 1 to 2, leader changed from 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3) to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 2 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:11.954506 1926 raft_consensus.cc:979] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:11.955060 1902 raft_consensus.cc:1067] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:11.956718 1636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e940099776f8417eaedcd9339857db23"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43176
I20260806 03:50:11.957275 1636 raft_consensus.cc:482] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:11.957621 1636 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.964210 1636 raft_consensus.cc:504] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.965988 1636 leader_election.cc:288] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 2 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:11.967358 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940099776f8417eaedcd9339857db23" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:11.967448 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940099776f8417eaedcd9339857db23" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:11.967938 1839 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.968200 1737 raft_consensus.cc:3032] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:11.968564 1737 raft_consensus.cc:729] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:11.969435 1737 consensus_queue.cc:260] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.970532 1737 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:11.973822 1839 raft_consensus.cc:2454] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 2.
I20260806 03:50:11.975039 1592 leader_election.cc:302] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters:
I20260806 03:50:11.975780 1950 raft_consensus.cc:2781] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:11.976225 1950 raft_consensus.cc:686] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:11.976732 1737 raft_consensus.cc:2454] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 2.
I20260806 03:50:11.976945 1950 consensus_queue.cc:237] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:11.984089 1489 catalog_manager.cc:5495] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 1 to 2, leader changed from a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 2 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:12.070116 1902 consensus_queue.cc:1035] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:12.077632 1926 raft_consensus.cc:979] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:12.078126 1981 raft_consensus.cc:1067] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:12.080727 1636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43176
I20260806 03:50:12.082589 1636 raft_consensus.cc:482] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:12.083001 1636 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:12.084000 1981 consensus_queue.cc:1035] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:12.091180 1636 raft_consensus.cc:504] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:12.092463 1637 raft_consensus.cc:1226] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Rejecting Update request from peer a3dcda65a7a141619bcc371f9a6d157e for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:12.095433 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:12.096573 1926 consensus_queue.cc:1046] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:12.097229 1636 leader_election.cc:288] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 2 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:12.098371 1902 raft_consensus.cc:3032] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:12.098737 1902 raft_consensus.cc:729] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:12.098881 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:12.099797 1902 consensus_queue.cc:260] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:12.100972 1902 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:12.113242 1737 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 2.
I20260806 03:50:12.115615 1590 leader_election.cc:302] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters: 2e2cb0a12b264e4190d87895271d3e8d
I20260806 03:50:12.116580 1950 raft_consensus.cc:2781] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:12.116976 1950 raft_consensus.cc:686] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:12.117717 1950 consensus_queue.cc:237] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:12.124676 1486 catalog_manager.cc:5495] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 1 to 2, leader changed from a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 2 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:12.204892 1945 raft_consensus.cc:979] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558: : Instructing follower a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:12.205410 1950 raft_consensus.cc:1067] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Signalling peer a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:12.206903 1737 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:44480
I20260806 03:50:12.207435 1737 raft_consensus.cc:482] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:12.207743 1737 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:12.213830 1737 raft_consensus.cc:504] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:12.215689 1737 leader_election.cc:288] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:12.216831 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:12.216758 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:12.217514 1839 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:12.217592 1637 raft_consensus.cc:3032] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:12.217975 1637 raft_consensus.cc:729] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:12.218828 1637 consensus_queue.cc:260] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:12.220257 1637 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:12.225461 1839 raft_consensus.cc:2454] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:12.226840 1694 leader_election.cc:302] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:12.227651 1902 raft_consensus.cc:2781] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:12.227676 1637 raft_consensus.cc:2454] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:12.228435 1902 raft_consensus.cc:686] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:12.229281 1902 consensus_queue.cc:237] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:12.236428 1486 catalog_manager.cc:5495] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 1 to 2, leader changed from 9edeaaf6653d49eab23847908f835558 (127.0.183.1) to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 2 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:12.249874 1737 raft_consensus.cc:1261] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.251320 1950 consensus_queue.cc:1035] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.261004 1838 raft_consensus.cc:1261] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.261503 1839 raft_consensus.cc:1261] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.263223 1950 consensus_queue.cc:1035] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.263155 1902 consensus_queue.cc:1035] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:12.267975 1737 raft_consensus.cc:1261] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.269467 1950 consensus_queue.cc:1035] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.276060 1637 raft_consensus.cc:1261] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.277714 1967 consensus_queue.cc:1035] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.278326 1636 raft_consensus.cc:1261] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.279898 1926 consensus_queue.cc:1035] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.287117 1838 raft_consensus.cc:1261] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.291601 1950 consensus_queue.cc:1035] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:12.291638 1741 raft_consensus.cc:1261] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.294528 1636 raft_consensus.cc:1261] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.297717 1838 raft_consensus.cc:1261] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.299109 1902 consensus_queue.cc:1035] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.304203 1967 consensus_queue.cc:1035] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.306200 1926 consensus_queue.cc:1035] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.351056 1838 raft_consensus.cc:1261] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.352851 1979 consensus_queue.cc:1035] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.361764 1637 raft_consensus.cc:1261] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.363722 1979 consensus_queue.cc:1035] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.502862 1741 raft_consensus.cc:1261] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.504367 1950 consensus_queue.cc:1035] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.513348 1838 raft_consensus.cc:1261] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.515404 1943 consensus_queue.cc:1035] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:12.623135 1637 raft_consensus.cc:1261] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.625948 1902 consensus_queue.cc:1035] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:12.636978 1838 raft_consensus.cc:1261] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.638608 1902 consensus_queue.cc:1035] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.665750 1838 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:12.674041 1838 raft_consensus.cc:1261] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.675835 1943 consensus_queue.cc:1035] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:12.688572 1741 raft_consensus.cc:1261] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:12.691679 2018 consensus_queue.cc:1035] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:14.748112 732 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260806 03:50:14.748916 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:14.774045 1637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:40674
I20260806 03:50:14.774679 1637 raft_consensus.cc:595] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Received request to transfer leadership to TS 2e2cb0a12b264e4190d87895271d3e8d
I20260806 03:50:14.777235 1637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:40674
I20260806 03:50:14.777760 1637 raft_consensus.cc:595] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Received request to transfer leadership to TS 2e2cb0a12b264e4190d87895271d3e8d
I20260806 03:50:14.778798 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260806 03:50:14.779296 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:14.785627 2028 raft_consensus.cc:979] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558: : Instructing follower 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:14.786092 2025 raft_consensus.cc:1067] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Signalling peer 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:14.787611 1838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:38742
I20260806 03:50:14.788153 1838 raft_consensus.cc:482] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:14.788473 1838 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:14.794055 1838 raft_consensus.cc:504] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:14.795934 1838 leader_election.cc:288] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 3 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:14.797058 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:14.797209 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:14.797677 1637 raft_consensus.cc:3032] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:50:14.797788 1741 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:14.797994 1637 raft_consensus.cc:729] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:14.798789 1637 consensus_queue.cc:260] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:14.799823 1637 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:14.804397 1741 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 3.
I20260806 03:50:14.805051 1637 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 3.
I20260806 03:50:14.805737 1793 leader_election.cc:302] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:14.806553 1967 raft_consensus.cc:2781] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:50:14.807673 1967 raft_consensus.cc:686] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Becoming Leader. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:14.808580 1967 consensus_queue.cc:237] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:14.817006 1489 catalog_manager.cc:5495] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d reported cstate change: term changed from 2 to 3, leader changed from 9edeaaf6653d49eab23847908f835558 (127.0.183.1) to 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3). New cstate: current_term: 3 leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:15.028277 2028 raft_consensus.cc:979] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558: : Instructing follower 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:15.028769 2025 raft_consensus.cc:1067] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Signalling peer 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:15.030388 1839 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:38742
I20260806 03:50:15.030974 1839 raft_consensus.cc:482] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:15.031286 1839 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:15.036607 1839 raft_consensus.cc:504] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:15.038313 1839 leader_election.cc:288] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 3 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:15.039243 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:15.039354 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:15.040050 1741 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:15.040004 1637 raft_consensus.cc:3032] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:50:15.040586 1637 raft_consensus.cc:729] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:15.041422 1637 consensus_queue.cc:260] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:15.042762 1637 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:15.048085 1741 raft_consensus.cc:2454] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 3.
I20260806 03:50:15.049461 1793 leader_election.cc:302] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:15.050326 1967 raft_consensus.cc:2781] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:50:15.050534 1637 raft_consensus.cc:2454] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 3.
I20260806 03:50:15.050887 1967 raft_consensus.cc:686] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Becoming Leader. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:15.051846 1967 consensus_queue.cc:237] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:15.059209 1489 catalog_manager.cc:5495] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d reported cstate change: term changed from 2 to 3, leader changed from 9edeaaf6653d49eab23847908f835558 (127.0.183.1) to 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3). New cstate: current_term: 3 leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:15.316062 1637 raft_consensus.cc:1261] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:15.317647 1967 consensus_queue.cc:1035] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:15.328992 1741 raft_consensus.cc:1261] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:15.330856 2044 consensus_queue.cc:1035] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:50:15.518141 1741 raft_consensus.cc:1261] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:15.519786 1967 consensus_queue.cc:1035] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:15.529987 1637 raft_consensus.cc:1261] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:15.532447 2044 consensus_queue.cc:1035] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:15.780488 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:15.786449 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:15.787041 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:16.788007 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:16.793658 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:16.794205 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:17.795684 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:17.802736 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:17.803404 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:18.804369 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:18.809861 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:18.810359 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:19.811359 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:19.817325 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:19.817799 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:20.818787 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:20.824368 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:20.824882 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:21.825978 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:21.831308 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:21.831817 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:22.832897 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:22.838402 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:22.838891 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:23.840240 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:23.846740 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:23.847270 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:24.884330 732 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260806 03:50:24.889474 1838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:56540
I20260806 03:50:24.890219 1838 raft_consensus.cc:595] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:24.900403 1838 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:56540
I20260806 03:50:24.901189 1838 raft_consensus.cc:595] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:24.912690 1741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:53318
I20260806 03:50:24.913254 1741 raft_consensus.cc:595] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:24.918342 1741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:53318
I20260806 03:50:24.918820 1741 raft_consensus.cc:595] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:24.923758 1741 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
mode: GRACEFUL
new_leader_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:53318
I20260806 03:50:24.924482 1741 raft_consensus.cc:595] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Received request to transfer leadership to TS 9edeaaf6653d49eab23847908f835558
I20260806 03:50:25.186918 2054 raft_consensus.cc:979] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.187460 2054 raft_consensus.cc:1067] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.189152 1637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43190
I20260806 03:50:25.189695 1637 raft_consensus.cc:482] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:25.190022 1637 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:25.198532 1637 raft_consensus.cc:504] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.200549 1637 leader_election.cc:288] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 4 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:25.201969 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:25.201969 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44b9197d3a3e4263b82a245d679b8361" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:25.202996 1741 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:25.202996 1838 raft_consensus.cc:3032] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:50:25.203547 1838 raft_consensus.cc:729] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:25.204551 1838 consensus_queue.cc:260] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.206125 1838 raft_consensus.cc:3037] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:25.212186 1741 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 4.
I20260806 03:50:25.213841 1590 leader_election.cc:302] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:25.214324 1838 raft_consensus.cc:2454] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 4.
I20260806 03:50:25.215088 2061 raft_consensus.cc:2781] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:50:25.216595 2061 raft_consensus.cc:686] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 4 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:25.217531 2061 consensus_queue.cc:237] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.225157 1491 catalog_manager.cc:5495] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 3 to 4, leader changed from 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 4 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:25.255234 2065 raft_consensus.cc:979] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.256893 2076 raft_consensus.cc:1067] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.259801 1636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43176
I20260806 03:50:25.260404 1636 raft_consensus.cc:482] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:25.261220 1636 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.262192 2076 raft_consensus.cc:979] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.262814 2065 raft_consensus.cc:1067] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.264631 1637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43176
I20260806 03:50:25.265265 1637 raft_consensus.cc:482] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:25.265659 1637 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.270573 1636 raft_consensus.cc:504] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.272933 1636 leader_election.cc:288] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 3 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:25.272939 1637 raft_consensus.cc:504] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.273859 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:25.274132 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dff61d968bfe463f8c94289437b27ba6" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:25.274417 1741 raft_consensus.cc:3032] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:50:25.274706 1838 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.274747 1741 raft_consensus.cc:729] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:25.276170 1637 leader_election.cc:288] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 3 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:25.275779 1741 consensus_queue.cc:260] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.276582 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:25.277076 1741 raft_consensus.cc:3037] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.277155 1839 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.280673 1838 raft_consensus.cc:2454] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 3.
I20260806 03:50:25.280776 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:25.281435 1737 raft_consensus.cc:3032] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:50:25.281917 1737 raft_consensus.cc:729] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:25.282332 1592 leader_election.cc:302] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters:
I20260806 03:50:25.283493 2061 raft_consensus.cc:2781] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:50:25.283964 2061 raft_consensus.cc:686] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:25.284819 2061 consensus_queue.cc:237] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.285808 1839 raft_consensus.cc:2454] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 3.
I20260806 03:50:25.286824 1592 leader_election.cc:302] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters:
I20260806 03:50:25.287817 2070 raft_consensus.cc:2781] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:50:25.282810 1737 consensus_queue.cc:260] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.289255 1741 raft_consensus.cc:2454] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 3.
I20260806 03:50:25.290230 1737 raft_consensus.cc:3037] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.297775 1737 raft_consensus.cc:2454] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 3.
I20260806 03:50:25.298391 2054 raft_consensus.cc:979] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.298913 2054 raft_consensus.cc:1067] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.300275 2076 raft_consensus.cc:979] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e: : Instructing follower 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.300995 2078 raft_consensus.cc:1067] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Signalling peer 9edeaaf6653d49eab23847908f835558 to start an election
I20260806 03:50:25.300155 1491 catalog_manager.cc:5495] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 2 to 3, leader changed from a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 3 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:25.301446 2070 raft_consensus.cc:686] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:25.302224 1637 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43190
I20260806 03:50:25.302881 1637 raft_consensus.cc:482] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:25.302763 2070 consensus_queue.cc:237] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.303172 1636 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
from {username='slave'} at 127.0.0.1:43176
I20260806 03:50:25.303334 1637 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:25.304073 1636 raft_consensus.cc:482] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:25.304669 1636 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.311975 1637 raft_consensus.cc:504] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.313897 1637 leader_election.cc:288] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 4 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:25.315034 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:25.315229 1839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:25.315874 1839 raft_consensus.cc:3032] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:50:25.315856 1737 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:25.316336 1839 raft_consensus.cc:729] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:25.317162 1839 consensus_queue.cc:260] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.318608 1839 raft_consensus.cc:3037] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:25.318470 1636 raft_consensus.cc:504] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.318470 1491 catalog_manager.cc:5495] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 2 to 3, leader changed from a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 3 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:25.320699 1636 leader_election.cc:288] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 3 election: Requested vote from peers a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:25.321805 1838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:25.322017 1741 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "9edeaaf6653d49eab23847908f835558" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:25.322379 1838 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.322629 1741 raft_consensus.cc:3032] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:50:25.322971 1741 raft_consensus.cc:729] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:25.323984 1737 raft_consensus.cc:2454] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 4.
I20260806 03:50:25.323802 1741 consensus_queue.cc:260] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.325335 1741 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:25.328344 1838 raft_consensus.cc:2454] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 3.
I20260806 03:50:25.328480 1590 leader_election.cc:302] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:25.330355 1592 leader_election.cc:302] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters:
I20260806 03:50:25.331138 1839 raft_consensus.cc:2454] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 4.
I20260806 03:50:25.331450 2080 raft_consensus.cc:2781] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:50:25.331929 2080 raft_consensus.cc:686] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 4 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:25.333007 2061 raft_consensus.cc:2781] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:50:25.332813 2080 consensus_queue.cc:237] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.336341 1741 raft_consensus.cc:2454] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9edeaaf6653d49eab23847908f835558 in term 3.
I20260806 03:50:25.339110 2061 raft_consensus.cc:686] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Becoming Leader. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:25.339962 2061 consensus_queue.cc:237] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:25.346165 1491 catalog_manager.cc:5495] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 3 to 4, leader changed from 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 4 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:25.357838 1491 catalog_manager.cc:5495] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 reported cstate change: term changed from 2 to 3, leader changed from a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2) to 9edeaaf6653d49eab23847908f835558 (127.0.183.1). New cstate: current_term: 3 leader_uuid: "9edeaaf6653d49eab23847908f835558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:25.635416 1838 raft_consensus.cc:1261] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:50:25.637014 2061 consensus_queue.cc:1035] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:50:25.648417 1741 raft_consensus.cc:1261] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:50:25.650154 2080 consensus_queue.cc:1035] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:50:25.690832 1838 raft_consensus.cc:1261] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:25.692144 2061 consensus_queue.cc:1035] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:25.703505 1741 raft_consensus.cc:1261] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:25.705564 2070 consensus_queue.cc:1035] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:25.843839 1838 raft_consensus.cc:1261] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:50:25.845376 2070 consensus_queue.cc:1035] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:50:25.855300 1741 raft_consensus.cc:1261] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:50:25.856925 2083 consensus_queue.cc:1035] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:50:25.886197 1838 raft_consensus.cc:1261] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:25.887519 2061 consensus_queue.cc:1035] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:25.890456 1741 raft_consensus.cc:1261] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:25.892095 2083 consensus_queue.cc:1035] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:25.908780 1741 raft_consensus.cc:1261] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:25.911020 2061 consensus_queue.cc:1035] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:25.926232 1838 raft_consensus.cc:1261] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Refusing update from remote peer 9edeaaf6653d49eab23847908f835558: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:25.927704 2061 consensus_queue.cc:1035] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:50:28.106060 732 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260806 03:50:28.107059 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:28.115094 1637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:40674
I20260806 03:50:28.116159 1637 raft_consensus.cc:595] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Received request to transfer leadership to TS a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:28.119361 1637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "507a2c08f1454493ac9114c065b91f1c"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:40674
I20260806 03:50:28.119922 1637 raft_consensus.cc:595] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Received request to transfer leadership to TS a3dcda65a7a141619bcc371f9a6d157e
I20260806 03:50:28.123369 1637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:40674
I20260806 03:50:28.123948 1637 raft_consensus.cc:595] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 4 LEADER]: Received request to transfer leadership to TS 2e2cb0a12b264e4190d87895271d3e8d
I20260806 03:50:28.126619 1637 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e940099776f8417eaedcd9339857db23"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:40674
I20260806 03:50:28.127166 1637 raft_consensus.cc:595] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Received request to transfer leadership to TS 2e2cb0a12b264e4190d87895271d3e8d
I20260806 03:50:28.128288 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 03:50:28.128868 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:29.181442 2115 raft_consensus.cc:979] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558: : Instructing follower 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:29.182111 2115 raft_consensus.cc:1067] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Signalling peer 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:29.185422 1837 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e940099776f8417eaedcd9339857db23"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:38742
I20260806 03:50:29.186211 1837 raft_consensus.cc:482] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:29.186851 1837 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:29.216771 2117 raft_consensus.cc:979] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558: : Instructing follower a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:29.287863 2114 raft_consensus.cc:1067] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Signalling peer a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:29.288875 1837 raft_consensus.cc:504] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.295667 2115 raft_consensus.cc:979] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558: : Instructing follower a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:29.296446 2115 raft_consensus.cc:1067] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Signalling peer a3dcda65a7a141619bcc371f9a6d157e to start an election
I20260806 03:50:29.301676 2119 raft_consensus.cc:482] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 9edeaaf6653d49eab23847908f835558)
I20260806 03:50:29.311640 2119 raft_consensus.cc:504] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.315083 1737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940099776f8417eaedcd9339857db23" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:29.316212 1737 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:29.317667 1637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e940099776f8417eaedcd9339857db23" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:29.317888 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:29.318387 1637 raft_consensus.cc:3032] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:50:29.318681 1637 raft_consensus.cc:729] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:29.320293 2119 leader_election.cc:288] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:29.319563 1637 consensus_queue.cc:260] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.321360 1736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c957f186dbc4ed3892239fac2ca12e4" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e" is_pre_election: true
I20260806 03:50:29.323943 1793 leader_election.cc:302] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d; no voters: 9edeaaf6653d49eab23847908f835558, a3dcda65a7a141619bcc371f9a6d157e
W20260806 03:50:29.325528 1590 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:34899, user_credentials={real_user=slave}} blocked reactor thread for 107852us
I20260806 03:50:29.330852 1736 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:44480
I20260806 03:50:29.332046 1736 raft_consensus.cc:482] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:29.332598 1837 leader_election.cc:288] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 3 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:29.332748 1736 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:29.326532 2119 raft_consensus.cc:2726] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:50:29.342260 1740 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "507a2c08f1454493ac9114c065b91f1c"
dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
from {username='slave'} at 127.0.0.1:44480
I20260806 03:50:29.343189 1740 raft_consensus.cc:482] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:29.343895 1740 raft_consensus.cc:3037] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:29.366966 1737 raft_consensus.cc:2454] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 3.
I20260806 03:50:29.375130 1793 leader_election.cc:302] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:29.369407 1740 raft_consensus.cc:504] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.376792 2119 raft_consensus.cc:2781] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:50:29.382580 1731 raft_consensus.cc:1226] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Rejecting Update request from peer 9edeaaf6653d49eab23847908f835558 for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:29.385335 2114 consensus_queue.cc:1046] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:29.386916 2114 raft_consensus.cc:3032] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:29.387341 2114 raft_consensus.cc:729] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:29.389058 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507a2c08f1454493ac9114c065b91f1c" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:29.388914 2114 consensus_queue.cc:260] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.391736 2114 raft_consensus.cc:3037] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:29.394245 1837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507a2c08f1454493ac9114c065b91f1c" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:29.395043 1837 raft_consensus.cc:3037] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:29.404392 2119 raft_consensus.cc:686] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Becoming Leader. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:29.408547 1740 leader_election.cc:288] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:29.410701 2119 consensus_queue.cc:237] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.414956 1837 raft_consensus.cc:2454] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
I20260806 03:50:29.419659 1694 leader_election.cc:302] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:29.421542 2126 raft_consensus.cc:2781] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:29.422194 2126 raft_consensus.cc:686] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:29.423691 2126 consensus_queue.cc:237] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.470652 1736 raft_consensus.cc:504] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.527181 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:29.528250 1635 raft_consensus.cc:3032] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:50:29.528398 1837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "225db8a5750f4a9d8eead9c679d013a8" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
I20260806 03:50:29.528697 1635 raft_consensus.cc:729] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:29.529314 1837 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:29.530139 1635 consensus_queue.cc:260] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.532639 1635 raft_consensus.cc:3037] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:29.548259 1736 leader_election.cc:288] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 3 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:29.583941 1837 raft_consensus.cc:2454] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 3.
I20260806 03:50:29.586539 1694 leader_election.cc:302] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, a3dcda65a7a141619bcc371f9a6d157e; no voters:
I20260806 03:50:29.588405 2125 raft_consensus.cc:2781] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:50:29.623240 1635 raft_consensus.cc:2454] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 3.
I20260806 03:50:29.639384 2125 raft_consensus.cc:686] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 3 LEADER]: Becoming Leader. State: Replica: a3dcda65a7a141619bcc371f9a6d157e, State: Running, Role: LEADER
I20260806 03:50:29.641124 2125 consensus_queue.cc:237] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
W20260806 03:50:29.759812 1769 debug-util.cc:398] Leaking SignalData structure 0x7b0800102be0 after lost signal to thread 735
W20260806 03:50:29.762578 1769 debug-util.cc:398] Leaking SignalData structure 0x7b0800101280 after lost signal to thread 1538
I20260806 03:50:29.770774 2126 raft_consensus.cc:482] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9edeaaf6653d49eab23847908f835558)
I20260806 03:50:29.771340 2124 raft_consensus.cc:482] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 9edeaaf6653d49eab23847908f835558)
I20260806 03:50:29.771920 2126 raft_consensus.cc:504] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.772089 2124 raft_consensus.cc:504] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:29.773494 1637 raft_consensus.cc:3037] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:50:29.774993 2126 leader_election.cc:288] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
W20260806 03:50:29.771126 1769 debug-util.cc:398] Leaking SignalData structure 0x7b080010e360 after lost signal to thread 1669
W20260806 03:50:29.777113 1769 debug-util.cc:398] Leaking SignalData structure 0x7b08000d08e0 after lost signal to thread 1777
I20260806 03:50:29.776865 1635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:29.777997 1842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f074e5cca98e4a4ea6eb6bfdc5185353" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
W20260806 03:50:29.778766 1769 debug-util.cc:398] Leaking SignalData structure 0x7b08001420a0 after lost signal to thread 1875
I20260806 03:50:29.779241 1633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "9edeaaf6653d49eab23847908f835558" is_pre_election: true
I20260806 03:50:29.777029 1837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "a3dcda65a7a141619bcc371f9a6d157e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d" is_pre_election: true
I20260806 03:50:29.782671 2124 leader_election.cc:288] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3:37601)
I20260806 03:50:29.783957 1687 leader_election.cc:302] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3dcda65a7a141619bcc371f9a6d157e; no voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558
I20260806 03:50:29.785360 1637 raft_consensus.cc:2454] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 3.
I20260806 03:50:29.785295 1687 leader_election.cc:302] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3dcda65a7a141619bcc371f9a6d157e; no voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558
I20260806 03:50:29.786870 2126 raft_consensus.cc:2726] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:50:29.787494 2126 raft_consensus.cc:2726] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:50:29.767854 1636 raft_consensus.cc:2454] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3dcda65a7a141619bcc371f9a6d157e in term 2.
W20260806 03:50:29.812832 2117 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.006s sys 0.009s
W20260806 03:50:29.813478 2117 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.596s user 0.007s sys 0.009s
I20260806 03:50:29.816159 1489 catalog_manager.cc:5495] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d reported cstate change: term changed from 2 to 3, leader changed from 9edeaaf6653d49eab23847908f835558 (127.0.183.1) to 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3). New cstate: current_term: 3 leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:29.822023 1491 catalog_manager.cc:5495] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 1 to 2, leader changed from 9edeaaf6653d49eab23847908f835558 (127.0.183.1) to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 2 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:29.825445 1491 catalog_manager.cc:5495] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e reported cstate change: term changed from 2 to 3, leader changed from 9edeaaf6653d49eab23847908f835558 (127.0.183.1) to a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2). New cstate: current_term: 3 leader_uuid: "a3dcda65a7a141619bcc371f9a6d157e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:29.904145 1636 raft_consensus.cc:1261] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:29.906018 2124 consensus_queue.cc:1035] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:29.921126 1838 raft_consensus.cc:1261] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:29.922984 2136 consensus_queue.cc:1035] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:29.939182 1741 raft_consensus.cc:1261] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:29.941807 2119 consensus_queue.cc:1035] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:50:30.001509 1636 raft_consensus.cc:1261] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:30.004411 2157 consensus_queue.cc:1035] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:50:30.032459 1636 raft_consensus.cc:1261] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:30.033825 2126 consensus_queue.cc:1035] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:50:30.047214 1838 raft_consensus.cc:1261] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Refusing update from remote peer a3dcda65a7a141619bcc371f9a6d157e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:50:30.048804 2136 consensus_queue.cc:1035] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 03:50:30.203581 1871 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad000 after lost signal to thread 735
W20260806 03:50:30.204780 1871 debug-util.cc:398] Leaking SignalData structure 0x7b08000ad7c0 after lost signal to thread 1538
W20260806 03:50:30.205808 1871 debug-util.cc:398] Leaking SignalData structure 0x7b08000bd320 after lost signal to thread 1669
W20260806 03:50:30.207008 1871 debug-util.cc:398] Leaking SignalData structure 0x7b08001a95e0 after lost signal to thread 1777
W20260806 03:50:30.208001 1871 debug-util.cc:398] Leaking SignalData structure 0x7b08001b40c0 after lost signal to thread 1875
I20260806 03:50:30.737083 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:30.748891 1636 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244"
dest_uuid: "9edeaaf6653d49eab23847908f835558"
mode: GRACEFUL
new_leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:40674
I20260806 03:50:30.749689 1636 raft_consensus.cc:595] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Received request to transfer leadership to TS 2e2cb0a12b264e4190d87895271d3e8d
I20260806 03:50:30.751281 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260806 03:50:30.752120 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:30.909181 2117 raft_consensus.cc:979] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558: : Instructing follower 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:30.909802 2117 raft_consensus.cc:1067] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Signalling peer 2e2cb0a12b264e4190d87895271d3e8d to start an election
I20260806 03:50:30.911868 1838 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244"
dest_uuid: "2e2cb0a12b264e4190d87895271d3e8d"
from {username='slave'} at 127.0.0.1:38742
I20260806 03:50:30.912556 1838 raft_consensus.cc:482] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:50:30.912936 1838 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:30.922353 1838 raft_consensus.cc:504] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:30.924569 1838 leader_election.cc:288] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 4 election: Requested vote from peers 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021), a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899)
I20260806 03:50:30.926940 1734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "a3dcda65a7a141619bcc371f9a6d157e"
I20260806 03:50:30.927688 1734 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:30.928455 1636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aae61aedb9448fd8efdb4ea28980244" candidate_uuid: "2e2cb0a12b264e4190d87895271d3e8d" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9edeaaf6653d49eab23847908f835558"
I20260806 03:50:30.929080 1636 raft_consensus.cc:3032] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:50:30.929382 1636 raft_consensus.cc:729] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9edeaaf6653d49eab23847908f835558, State: Running, Role: LEADER
I20260806 03:50:30.930186 1636 consensus_queue.cc:260] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:30.931630 1636 raft_consensus.cc:3037] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:50:30.941252 1636 raft_consensus.cc:2454] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 4.
I20260806 03:50:30.942126 1734 raft_consensus.cc:2454] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e2cb0a12b264e4190d87895271d3e8d in term 4.
I20260806 03:50:30.942926 1791 leader_election.cc:302] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e2cb0a12b264e4190d87895271d3e8d, 9edeaaf6653d49eab23847908f835558; no voters:
I20260806 03:50:30.944285 2119 raft_consensus.cc:2781] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:50:30.944811 2119 raft_consensus.cc:686] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 LEADER]: Becoming Leader. State: Replica: 2e2cb0a12b264e4190d87895271d3e8d, State: Running, Role: LEADER
I20260806 03:50:30.945906 2119 consensus_queue.cc:237] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } }
I20260806 03:50:30.958228 1491 catalog_manager.cc:5495] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d reported cstate change: term changed from 3 to 4, leader changed from 9edeaaf6653d49eab23847908f835558 (127.0.183.1) to 2e2cb0a12b264e4190d87895271d3e8d (127.0.183.3). New cstate: current_term: 4 leader_uuid: "2e2cb0a12b264e4190d87895271d3e8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e2cb0a12b264e4190d87895271d3e8d" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 37601 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:31.452143 1636 raft_consensus.cc:1261] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:50:31.453660 2119 consensus_queue.cc:1035] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9edeaaf6653d49eab23847908f835558" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39021 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:50:31.475082 1734 raft_consensus.cc:1261] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Refusing update from remote peer 2e2cb0a12b264e4190d87895271d3e8d: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:50:31.481825 2119 consensus_queue.cc:1035] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3dcda65a7a141619bcc371f9a6d157e" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 34899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:50:31.753407 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:31.763307 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:31.764034 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:32.765370 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:32.776497 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:32.777216 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:33.778301 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:33.784195 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:33.784685 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:34.838827 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:34.844918 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:34.845419 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:50:35.317111 1663 debug-util.cc:398] Leaking SignalData structure 0x7b080014bee0 after lost signal to thread 735
W20260806 03:50:35.318262 1663 debug-util.cc:398] Leaking SignalData structure 0x7b08001a9cc0 after lost signal to thread 1538
W20260806 03:50:35.319640 1663 debug-util.cc:398] Leaking SignalData structure 0x7b0800093180 after lost signal to thread 1669
W20260806 03:50:35.320757 1663 debug-util.cc:398] Leaking SignalData structure 0x7b08001c11c0 after lost signal to thread 1777
W20260806 03:50:35.321807 1663 debug-util.cc:398] Leaking SignalData structure 0x7b08001c0be0 after lost signal to thread 1875
I20260806 03:50:35.846503 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:35.853233 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:35.853837 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:36.854831 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:36.862416 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:36.862987 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:37.864616 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:37.871757 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:37.872227 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:38.873216 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:50:38.879115 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260806 03:50:38.879601 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:50:39.914176 732 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260806 03:50:39.914737 732 tablet_server.cc:178] TabletServer@127.0.183.1:0 shutting down...
I20260806 03:50:39.963294 732 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:50:39.964124 732 tablet_replica.cc:330] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.965061 732 raft_consensus.cc:2229] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:39.965696 732 raft_consensus.cc:2258] T 225db8a5750f4a9d8eead9c679d013a8 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:39.967986 732 tablet_replica.cc:330] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.968665 732 raft_consensus.cc:2229] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:50:39.969779 732 raft_consensus.cc:2258] T 9c957f186dbc4ed3892239fac2ca12e4 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:39.971997 732 tablet_replica.cc:330] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.972568 732 raft_consensus.cc:2229] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:39.973004 732 raft_consensus.cc:2258] T 46f581f68b7a4e6187491b09f936f342 P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:39.974967 732 tablet_replica.cc:330] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.975528 732 raft_consensus.cc:2229] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:39.976096 732 raft_consensus.cc:2258] T 09de95e25a5b47bbb0e59c671908c7ab P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260806 03:50:39.977566 1687 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 6 similar messages]
I20260806 03:50:39.978961 732 tablet_replica.cc:330] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.979692 732 raft_consensus.cc:2229] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:39.980286 732 raft_consensus.cc:2258] T 507a2c08f1454493ac9114c065b91f1c P 9edeaaf6653d49eab23847908f835558 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260806 03:50:39.981402 1687 consensus_peers.cc:487] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e -> Peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Couldn't send request to peer 9edeaaf6653d49eab23847908f835558. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:39.982841 732 tablet_replica.cc:330] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.983430 732 raft_consensus.cc:2229] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:39.983901 732 raft_consensus.cc:2258] T 8aae61aedb9448fd8efdb4ea28980244 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:39.985842 732 tablet_replica.cc:330] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.986387 732 raft_consensus.cc:2229] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:50:39.987332 732 raft_consensus.cc:2258] T f074e5cca98e4a4ea6eb6bfdc5185353 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:39.989215 732 tablet_replica.cc:330] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.989738 732 raft_consensus.cc:2229] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:39.990226 732 raft_consensus.cc:2258] T e940099776f8417eaedcd9339857db23 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:39.992125 732 tablet_replica.cc:330] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.992641 732 raft_consensus.cc:2229] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:50:39.993623 732 raft_consensus.cc:2258] T dff61d968bfe463f8c94289437b27ba6 P 9edeaaf6653d49eab23847908f835558 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:39.995448 732 tablet_replica.cc:330] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558: stopping tablet replica
I20260806 03:50:39.995941 732 raft_consensus.cc:2229] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:50:39.997048 732 raft_consensus.cc:2258] T 44b9197d3a3e4263b82a245d679b8361 P 9edeaaf6653d49eab23847908f835558 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.050951 732 tablet_server.cc:194] TabletServer@127.0.183.1:0 shutdown complete.
I20260806 03:50:40.078101 732 tablet_server.cc:178] TabletServer@127.0.183.2:0 shutting down...
I20260806 03:50:40.117169 732 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:50:40.117964 732 tablet_replica.cc:330] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.118796 732 raft_consensus.cc:2229] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.119437 732 raft_consensus.cc:2258] T 8aae61aedb9448fd8efdb4ea28980244 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.122412 732 tablet_replica.cc:330] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.123059 732 raft_consensus.cc:2229] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.123610 732 raft_consensus.cc:2258] T dff61d968bfe463f8c94289437b27ba6 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.125783 732 tablet_replica.cc:330] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.126407 732 raft_consensus.cc:2229] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.127218 732 raft_consensus.cc:2258] T 44b9197d3a3e4263b82a245d679b8361 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.129405 732 tablet_replica.cc:330] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.129995 732 raft_consensus.cc:2229] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.130564 732 raft_consensus.cc:2258] T 9c957f186dbc4ed3892239fac2ca12e4 P a3dcda65a7a141619bcc371f9a6d157e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.132730 732 tablet_replica.cc:330] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.133354 732 raft_consensus.cc:2229] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:50:40.134383 732 raft_consensus.cc:2258] T 507a2c08f1454493ac9114c065b91f1c P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.136787 732 tablet_replica.cc:330] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.137428 732 raft_consensus.cc:2229] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:50:40.138453 732 raft_consensus.cc:2258] T 09de95e25a5b47bbb0e59c671908c7ab P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.140475 732 tablet_replica.cc:330] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.140995 732 raft_consensus.cc:2229] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.141462 732 raft_consensus.cc:2258] T 46f581f68b7a4e6187491b09f936f342 P a3dcda65a7a141619bcc371f9a6d157e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.143823 732 tablet_replica.cc:330] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.144524 732 raft_consensus.cc:2229] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:50:40.146328 732 raft_consensus.cc:2258] T 225db8a5750f4a9d8eead9c679d013a8 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.149878 732 tablet_replica.cc:330] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.150756 732 raft_consensus.cc:2229] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Raft consensus shutting down.
W20260806 03:50:40.150725 1793 consensus_peers.cc:487] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d -> Peer a3dcda65a7a141619bcc371f9a6d157e (127.0.183.2:34899): Couldn't send request to peer a3dcda65a7a141619bcc371f9a6d157e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:40.151576 732 raft_consensus.cc:2258] T e940099776f8417eaedcd9339857db23 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Raft consensus is shut down!
W20260806 03:50:40.153358 1791 consensus_peers.cc:487] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d -> Peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Couldn't send request to peer 9edeaaf6653d49eab23847908f835558. Status: Network error: Client connection negotiation failed: client connection to 127.0.183.1:39021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:40.153968 732 tablet_replica.cc:330] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e: stopping tablet replica
I20260806 03:50:40.154510 732 raft_consensus.cc:2229] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.154966 732 raft_consensus.cc:2258] T f074e5cca98e4a4ea6eb6bfdc5185353 P a3dcda65a7a141619bcc371f9a6d157e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.206843 732 tablet_server.cc:194] TabletServer@127.0.183.2:0 shutdown complete.
I20260806 03:50:40.233132 732 tablet_server.cc:178] TabletServer@127.0.183.3:0 shutting down...
W20260806 03:50:40.272006 1791 consensus_peers.cc:487] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d -> Peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Couldn't send request to peer 9edeaaf6653d49eab23847908f835558. Status: Network error: Client connection negotiation failed: client connection to 127.0.183.1:39021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260806 03:50:40.277215 1791 consensus_peers.cc:487] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d -> Peer 9edeaaf6653d49eab23847908f835558 (127.0.183.1:39021): Couldn't send request to peer 9edeaaf6653d49eab23847908f835558. Status: Network error: Client connection negotiation failed: client connection to 127.0.183.1:39021: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:40.278764 732 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260806 03:50:40.279517 732 tablet_replica.cc:330] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.280428 732 raft_consensus.cc:2229] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.281011 732 raft_consensus.cc:2258] T 507a2c08f1454493ac9114c065b91f1c P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.283360 732 tablet_replica.cc:330] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.283967 732 raft_consensus.cc:2229] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.284531 732 raft_consensus.cc:2258] T 225db8a5750f4a9d8eead9c679d013a8 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.286520 732 tablet_replica.cc:330] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.287089 732 raft_consensus.cc:2229] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.287578 732 raft_consensus.cc:2258] T dff61d968bfe463f8c94289437b27ba6 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.289484 732 tablet_replica.cc:330] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.290023 732 raft_consensus.cc:2229] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 LEADER]: Raft consensus shutting down.
I20260806 03:50:40.290978 732 raft_consensus.cc:2258] T 46f581f68b7a4e6187491b09f936f342 P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.293211 732 tablet_replica.cc:330] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.293877 732 raft_consensus.cc:2229] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.294417 732 raft_consensus.cc:2258] T 09de95e25a5b47bbb0e59c671908c7ab P 2e2cb0a12b264e4190d87895271d3e8d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.296415 732 tablet_replica.cc:330] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.296979 732 raft_consensus.cc:2229] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.297487 732 raft_consensus.cc:2258] T 44b9197d3a3e4263b82a245d679b8361 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.299355 732 tablet_replica.cc:330] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.299896 732 raft_consensus.cc:2229] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.300494 732 raft_consensus.cc:2258] T f074e5cca98e4a4ea6eb6bfdc5185353 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.302395 732 tablet_replica.cc:330] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.302928 732 raft_consensus.cc:2229] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 LEADER]: Raft consensus shutting down.
I20260806 03:50:40.303790 732 raft_consensus.cc:2258] T e940099776f8417eaedcd9339857db23 P 2e2cb0a12b264e4190d87895271d3e8d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.305711 732 tablet_replica.cc:330] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.306244 732 raft_consensus.cc:2229] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Raft consensus shutting down.
I20260806 03:50:40.306722 732 raft_consensus.cc:2258] T 9c957f186dbc4ed3892239fac2ca12e4 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.308733 732 tablet_replica.cc:330] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d: stopping tablet replica
I20260806 03:50:40.309222 732 raft_consensus.cc:2229] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 LEADER]: Raft consensus shutting down.
I20260806 03:50:40.310127 732 raft_consensus.cc:2258] T 8aae61aedb9448fd8efdb4ea28980244 P 2e2cb0a12b264e4190d87895271d3e8d [term 4 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.360733 732 tablet_server.cc:194] TabletServer@127.0.183.3:0 shutdown complete.
I20260806 03:50:40.384778 732 master.cc:537] Master@127.0.183.62:38901 shutting down...
I20260806 03:50:40.416386 732 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 1 LEADER]: Raft consensus shutting down.
I20260806 03:50:40.417081 732 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:50:40.417412 732 tablet_replica.cc:330] T 00000000000000000000000000000000 P 61c6e08dac694b38b8dc3842649f3893: stopping tablet replica
I20260806 03:50:40.439461 732 master.cc:552] Master@127.0.183.62:38901 shutdown complete.
2e2cb0a12b264e4190d87895271d3e8d 1
9edeaaf6653d49eab23847908f835558 5
a3dcda65a7a141619bcc371f9a6d157e 4
2e2cb0a12b264e4190d87895271d3e8d 3
9edeaaf6653d49eab23847908f835558 3
a3dcda65a7a141619bcc371f9a6d157e 4
2e2cb0a12b264e4190d87895271d3e8d 1
9edeaaf6653d49eab23847908f835558 8
a3dcda65a7a141619bcc371f9a6d157e 1
2e2cb0a12b264e4190d87895271d3e8d 3
9edeaaf6653d49eab23847908f835558 4
a3dcda65a7a141619bcc371f9a6d157e 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30874 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260806 03:50:40.489846 732 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.0.183.62:40035
I20260806 03:50:40.491170 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:50:40.498157 2203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:40.498265 2202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:40.500130 2205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:50:40.501436 732 server_base.cc:1034] running on GCE node
I20260806 03:50:40.502854 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:50:40.503098 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:50:40.503268 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988240503247 us; error 0 us; skew 500 ppm
I20260806 03:50:40.503839 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:50:40.511911 732 webserver.cc:460] Webserver started at http://127.0.183.62:42509/ using document root <none> and password file <none>
I20260806 03:50:40.512598 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:50:40.512820 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:50:40.513149 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:50:40.514564 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/master-0-root/instance:
uuid: "e1fd51235035487c913ffce700debedb"
format_stamp: "Formatted at 2026-08-06 03:50:40 on dist-test-slave-kvfs"
I20260806 03:50:40.519790 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 03:50:40.523505 2213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:40.524609 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 03:50:40.524893 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/master-0-root
uuid: "e1fd51235035487c913ffce700debedb"
format_stamp: "Formatted at 2026-08-06 03:50:40 on dist-test-slave-kvfs"
I20260806 03:50:40.525146 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:50:40.557690 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:50:40.559212 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:50:40.615739 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.62:40035
I20260806 03:50:40.615842 2283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.62:40035 every 8 connection(s)
I20260806 03:50:40.620599 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:40.634229 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Bootstrap starting.
I20260806 03:50:40.640221 2284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:40.645923 2284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: No bootstrap required, opened a new log
I20260806 03:50:40.648802 2284 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fd51235035487c913ffce700debedb" member_type: VOTER }
I20260806 03:50:40.649420 2284 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:40.649752 2284 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1fd51235035487c913ffce700debedb, State: Initialized, Role: FOLLOWER
I20260806 03:50:40.650576 2284 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fd51235035487c913ffce700debedb" member_type: VOTER }
I20260806 03:50:40.651132 2284 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 03:50:40.651388 2284 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 03:50:40.651710 2284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:40.657749 2284 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fd51235035487c913ffce700debedb" member_type: VOTER }
I20260806 03:50:40.658398 2284 leader_election.cc:302] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1fd51235035487c913ffce700debedb; no voters:
I20260806 03:50:40.659973 2284 leader_election.cc:288] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 03:50:40.660584 2288 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:40.662134 2288 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [term 1 LEADER]: Becoming Leader. State: Replica: e1fd51235035487c913ffce700debedb, State: Running, Role: LEADER
I20260806 03:50:40.662910 2288 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fd51235035487c913ffce700debedb" member_type: VOTER }
I20260806 03:50:40.663525 2284 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 03:50:40.668658 2289 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1fd51235035487c913ffce700debedb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fd51235035487c913ffce700debedb" member_type: VOTER } }
I20260806 03:50:40.669474 2289 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [sys.catalog]: This master's current role is: LEADER
I20260806 03:50:40.669624 2291 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1fd51235035487c913ffce700debedb. Latest consensus state: current_term: 1 leader_uuid: "e1fd51235035487c913ffce700debedb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1fd51235035487c913ffce700debedb" member_type: VOTER } }
I20260806 03:50:40.670611 2291 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb [sys.catalog]: This master's current role is: LEADER
I20260806 03:50:40.676955 2298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 03:50:40.686018 2298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 03:50:40.687237 732 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260806 03:50:40.697046 2298 catalog_manager.cc:1347] Generated new cluster ID: 0194017e79b840f08605a7482a7751e5
I20260806 03:50:40.697438 2298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 03:50:40.723405 2298 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 03:50:40.725360 2298 catalog_manager.cc:1504] Loading token signing keys...
I20260806 03:50:40.738694 2298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Generated new TSK 0
I20260806 03:50:40.739527 2298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 03:50:40.754544 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:50:40.761662 2314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:40.763636 2315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:40.765591 2317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:50:40.766278 732 server_base.cc:1034] running on GCE node
I20260806 03:50:40.767555 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:50:40.767771 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:50:40.767966 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988240767950 us; error 0 us; skew 500 ppm
I20260806 03:50:40.768618 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:50:40.771449 732 webserver.cc:460] Webserver started at http://127.0.183.1:35181/ using document root <none> and password file <none>
I20260806 03:50:40.772003 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:50:40.772274 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:50:40.772581 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:50:40.774014 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root/instance:
uuid: "dff94100ffc5485f92cbe6a85addf0b2"
format_stamp: "Formatted at 2026-08-06 03:50:40 on dist-test-slave-kvfs"
I20260806 03:50:40.779177 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260806 03:50:40.783030 2326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:40.783849 732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 03:50:40.784252 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root
uuid: "dff94100ffc5485f92cbe6a85addf0b2"
format_stamp: "Formatted at 2026-08-06 03:50:40 on dist-test-slave-kvfs"
I20260806 03:50:40.784539 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:50:40.799649 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:50:40.801203 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:50:40.803161 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:50:40.805889 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:50:40.806138 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:40.806417 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:50:40.806615 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:40.862288 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.1:39643
I20260806 03:50:40.862426 2413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.1:39643 every 8 connection(s)
I20260806 03:50:40.868320 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:50:40.878111 2418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:40.880491 2419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:40.887317 2422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:50:40.889072 732 server_base.cc:1034] running on GCE node
I20260806 03:50:40.890527 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:50:40.890776 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:50:40.890964 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988240890949 us; error 0 us; skew 500 ppm
I20260806 03:50:40.891553 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:50:40.894333 2414 heartbeater.cc:347] Connected to a master server at 127.0.183.62:40035
I20260806 03:50:40.894850 2414 heartbeater.cc:464] Registering TS with master...
I20260806 03:50:40.894863 732 webserver.cc:460] Webserver started at http://127.0.183.2:45835/ using document root <none> and password file <none>
I20260806 03:50:40.895701 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:50:40.895767 2414 heartbeater.cc:511] Master 127.0.183.62:40035 requested a full tablet report, sending...
I20260806 03:50:40.895975 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:50:40.896422 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:50:40.897899 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/instance:
uuid: "608e57d72aae496aaf8613622153aa0c"
format_stamp: "Formatted at 2026-08-06 03:50:40 on dist-test-slave-kvfs"
I20260806 03:50:40.898348 2236 ts_manager.cc:194] Registered new tserver with Master: dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:50:40.900592 2236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33762
I20260806 03:50:40.903661 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 03:50:40.908268 2428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:40.909353 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 03:50:40.909719 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root
uuid: "608e57d72aae496aaf8613622153aa0c"
format_stamp: "Formatted at 2026-08-06 03:50:40 on dist-test-slave-kvfs"
I20260806 03:50:40.910301 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:50:40.937294 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:50:40.938982 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:50:40.941104 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:50:40.944422 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:50:40.944679 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:40.944980 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:50:40.945153 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:40.996971 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.2:45689
I20260806 03:50:40.997098 2516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.2:45689 every 8 connection(s)
I20260806 03:50:41.002668 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:50:41.011322 2522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:41.012385 2523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:50:41.016009 2527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:50:41.016613 732 server_base.cc:1034] running on GCE node
I20260806 03:50:41.018735 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:50:41.019178 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:50:41.019428 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988241019403 us; error 0 us; skew 500 ppm
I20260806 03:50:41.020159 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:50:41.020670 2517 heartbeater.cc:347] Connected to a master server at 127.0.183.62:40035
I20260806 03:50:41.021080 2517 heartbeater.cc:464] Registering TS with master...
I20260806 03:50:41.021974 2517 heartbeater.cc:511] Master 127.0.183.62:40035 requested a full tablet report, sending...
I20260806 03:50:41.024431 2236 ts_manager.cc:194] Registered new tserver with Master: 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:41.024982 732 webserver.cc:460] Webserver started at http://127.0.183.3:43793/ using document root <none> and password file <none>
I20260806 03:50:41.025564 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:50:41.025767 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:50:41.026108 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:50:41.026520 2236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33778
I20260806 03:50:41.027746 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/instance:
uuid: "fe3d17f4ad984b9088a268d58d87469c"
format_stamp: "Formatted at 2026-08-06 03:50:41 on dist-test-slave-kvfs"
I20260806 03:50:41.032934 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 03:50:41.037192 2533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:41.038342 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 03:50:41.038915 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root
uuid: "fe3d17f4ad984b9088a268d58d87469c"
format_stamp: "Formatted at 2026-08-06 03:50:41 on dist-test-slave-kvfs"
I20260806 03:50:41.039250 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:50:41.059397 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:50:41.060910 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:50:41.062743 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:50:41.067173 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:50:41.067425 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:41.067727 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:50:41.067902 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:50:41.118026 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.3:46407
I20260806 03:50:41.118131 2620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.3:46407 every 8 connection(s)
I20260806 03:50:41.136833 2622 heartbeater.cc:347] Connected to a master server at 127.0.183.62:40035
I20260806 03:50:41.137308 2622 heartbeater.cc:464] Registering TS with master...
I20260806 03:50:41.138101 2622 heartbeater.cc:511] Master 127.0.183.62:40035 requested a full tablet report, sending...
I20260806 03:50:41.140561 2236 ts_manager.cc:194] Registered new tserver with Master: fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:41.140681 732 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.019242285s
I20260806 03:50:41.142690 2236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33786
I20260806 03:50:41.143656 732 test_util.cc:268] Using random seed: -894453942
I20260806 03:50:41.175135 2236 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33798:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 03:50:41.182411 2236 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 03:50:41.429325 2473 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc4c26e48e41e3acb2121cf91b6df1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 03:50:41.431016 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc4c26e48e41e3acb2121cf91b6df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.433241 2472 tablet_service.cc:1467] Processing CreateTablet for tablet ac682b8be46c4b6599610372a4c8c727 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 03:50:41.434814 2472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac682b8be46c4b6599610372a4c8c727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.437393 2469 tablet_service.cc:1467] Processing CreateTablet for tablet 460a21786a7449039551dc0f1ffd04e2 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 03:50:41.438046 2571 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc4c26e48e41e3acb2121cf91b6df1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 03:50:41.438972 2469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460a21786a7449039551dc0f1ffd04e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.439399 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc4c26e48e41e3acb2121cf91b6df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.440582 2564 tablet_service.cc:1467] Processing CreateTablet for tablet e9f040dd271944b9bf08554a90afa940 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 03:50:41.441115 2563 tablet_service.cc:1467] Processing CreateTablet for tablet dad99dac270543b19675707d75149e78 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 03:50:41.441708 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9fc024596a478e80a8be24774725e5 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 03:50:41.442552 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad99dac270543b19675707d75149e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.443624 2566 tablet_service.cc:1467] Processing CreateTablet for tablet 47ed8dc77b144ba08169a0090b787536 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 03:50:41.444265 2570 tablet_service.cc:1467] Processing CreateTablet for tablet ac682b8be46c4b6599610372a4c8c727 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 03:50:41.445154 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ed8dc77b144ba08169a0090b787536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.445924 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac682b8be46c4b6599610372a4c8c727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.446723 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 8390916b6ac84b59b3943bf835a36a4c (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 03:50:41.448174 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8390916b6ac84b59b3943bf835a36a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.438994 2569 tablet_service.cc:1467] Processing CreateTablet for tablet 460a21786a7449039551dc0f1ffd04e2 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 03:50:41.453567 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460a21786a7449039551dc0f1ffd04e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.453421 2567 tablet_service.cc:1467] Processing CreateTablet for tablet d93b8d7e5d334d3dac131b9a4c49b103 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 03:50:41.455878 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 42ea7d4af5d44cab89bf24f15376456e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 03:50:41.457314 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ea7d4af5d44cab89bf24f15376456e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.459015 2465 tablet_service.cc:1467] Processing CreateTablet for tablet e9f040dd271944b9bf08554a90afa940 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 03:50:41.459299 2468 tablet_service.cc:1467] Processing CreateTablet for tablet 8390916b6ac84b59b3943bf835a36a4c (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 03:50:41.461323 2466 tablet_service.cc:1467] Processing CreateTablet for tablet 47ed8dc77b144ba08169a0090b787536 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 03:50:41.463333 2464 tablet_service.cc:1467] Processing CreateTablet for tablet dad99dac270543b19675707d75149e78 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 03:50:41.461447 2467 tablet_service.cc:1467] Processing CreateTablet for tablet d93b8d7e5d334d3dac131b9a4c49b103 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 03:50:41.465900 2465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f040dd271944b9bf08554a90afa940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.466472 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f040dd271944b9bf08554a90afa940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.467494 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 6796c2cfd90d4dc9ab9759b7845f0fd0 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 03:50:41.468667 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93b8d7e5d334d3dac131b9a4c49b103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.469579 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6796c2cfd90d4dc9ab9759b7845f0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.470386 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9fc024596a478e80a8be24774725e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.468729 2468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8390916b6ac84b59b3943bf835a36a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.474598 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad99dac270543b19675707d75149e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.478672 2467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93b8d7e5d334d3dac131b9a4c49b103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.484494 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ed8dc77b144ba08169a0090b787536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.490613 2463 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9fc024596a478e80a8be24774725e5 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 03:50:41.494175 2461 tablet_service.cc:1467] Processing CreateTablet for tablet 42ea7d4af5d44cab89bf24f15376456e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 03:50:41.497056 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9fc024596a478e80a8be24774725e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.498205 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 750d734a677a4861bca565b69d29396e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 03:50:41.500380 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750d734a677a4861bca565b69d29396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.500526 2461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ea7d4af5d44cab89bf24f15376456e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.515125 2367 tablet_service.cc:1467] Processing CreateTablet for tablet 24cc4c26e48e41e3acb2121cf91b6df1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260806 03:50:41.516762 2367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24cc4c26e48e41e3acb2121cf91b6df1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.521893 2366 tablet_service.cc:1467] Processing CreateTablet for tablet ac682b8be46c4b6599610372a4c8c727 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260806 03:50:41.523363 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac682b8be46c4b6599610372a4c8c727. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.529793 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 460a21786a7449039551dc0f1ffd04e2 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260806 03:50:41.531610 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 460a21786a7449039551dc0f1ffd04e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.535048 2364 tablet_service.cc:1467] Processing CreateTablet for tablet 8390916b6ac84b59b3943bf835a36a4c (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260806 03:50:41.537344 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8390916b6ac84b59b3943bf835a36a4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.574867 2642 tablet_bootstrap.cc:492] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:41.575409 2570 tablet_service.cc:1467] Processing CreateTablet for tablet bd8a4475fdfd488b8b2a5456043cfbc5 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 03:50:41.577334 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8a4475fdfd488b8b2a5456043cfbc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.584926 2563 tablet_service.cc:1467] Processing CreateTablet for tablet 14fa621f65da481d96d569096ded0a35 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 03:50:41.586926 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fa621f65da481d96d569096ded0a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.598124 2363 tablet_service.cc:1467] Processing CreateTablet for tablet d93b8d7e5d334d3dac131b9a4c49b103 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260806 03:50:41.599812 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93b8d7e5d334d3dac131b9a4c49b103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.606043 2359 tablet_service.cc:1467] Processing CreateTablet for tablet 47ed8dc77b144ba08169a0090b787536 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260806 03:50:41.607779 2359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ed8dc77b144ba08169a0090b787536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.609390 2358 tablet_service.cc:1467] Processing CreateTablet for tablet e9f040dd271944b9bf08554a90afa940 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260806 03:50:41.612083 2357 tablet_service.cc:1467] Processing CreateTablet for tablet dad99dac270543b19675707d75149e78 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260806 03:50:41.614562 2355 tablet_service.cc:1467] Processing CreateTablet for tablet 42ea7d4af5d44cab89bf24f15376456e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260806 03:50:41.609390 2356 tablet_service.cc:1467] Processing CreateTablet for tablet 1f9fc024596a478e80a8be24774725e5 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260806 03:50:41.620481 2356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f9fc024596a478e80a8be24774725e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.624450 2357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dad99dac270543b19675707d75149e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.632555 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ea7d4af5d44cab89bf24f15376456e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.634361 2358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f040dd271944b9bf08554a90afa940. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:50:41.681072 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:41.689028 2642 tablet_bootstrap.cc:654] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:41.692418 2563 tablet_service.cc:1467] Processing CreateTablet for tablet 223d3e4f95dc4407a7abb4d09b0cceb7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 03:50:41.693862 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223d3e4f95dc4407a7abb4d09b0cceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.696535 2355 tablet_service.cc:1467] Processing CreateTablet for tablet 14fa621f65da481d96d569096ded0a35 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 03:50:41.697945 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fa621f65da481d96d569096ded0a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.721009 2644 tablet_bootstrap.cc:492] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:41.724716 2570 tablet_service.cc:1467] Processing CreateTablet for tablet f3ae6cf9eba9410d8307ed1da6a62ea6 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 03:50:41.726307 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ae6cf9eba9410d8307ed1da6a62ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.737399 2644 tablet_bootstrap.cc:654] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:41.744663 2467 tablet_service.cc:1467] Processing CreateTablet for tablet bd8a4475fdfd488b8b2a5456043cfbc5 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 03:50:41.747084 2467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8a4475fdfd488b8b2a5456043cfbc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.748940 2465 tablet_service.cc:1467] Processing CreateTablet for tablet 750d734a677a4861bca565b69d29396e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 03:50:41.750535 2465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750d734a677a4861bca565b69d29396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.751163 2469 tablet_service.cc:1467] Processing CreateTablet for tablet 6796c2cfd90d4dc9ab9759b7845f0fd0 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 03:50:41.759276 2561 tablet_service.cc:1467] Processing CreateTablet for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 03:50:41.761663 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.764968 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 04b1317a33c34953b84b5a8c6598368f (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 03:50:41.766675 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b1317a33c34953b84b5a8c6598368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.772454 2643 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:41.777631 2642 tablet_bootstrap.cc:492] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:41.778184 2642 ts_tablet_manager.cc:1397] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.204s user 0.011s sys 0.008s
I20260806 03:50:41.781409 2357 tablet_service.cc:1467] Processing CreateTablet for tablet bd8a4475fdfd488b8b2a5456043cfbc5 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260806 03:50:41.782964 2357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8a4475fdfd488b8b2a5456043cfbc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.784202 2469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6796c2cfd90d4dc9ab9759b7845f0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.784751 2463 tablet_service.cc:1467] Processing CreateTablet for tablet 14fa621f65da481d96d569096ded0a35 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260806 03:50:41.786142 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fa621f65da481d96d569096ded0a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.792212 2468 tablet_service.cc:1467] Processing CreateTablet for tablet f3ae6cf9eba9410d8307ed1da6a62ea6 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 03:50:41.793684 2468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ae6cf9eba9410d8307ed1da6a62ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.797381 2461 tablet_service.cc:1467] Processing CreateTablet for tablet 223d3e4f95dc4407a7abb4d09b0cceb7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 03:50:41.797758 2569 tablet_service.cc:1467] Processing CreateTablet for tablet b91e8fd3fd1340d2bd8db09b77bd7b0f (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 03:50:41.799700 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91e8fd3fd1340d2bd8db09b77bd7b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.801051 2464 tablet_service.cc:1467] Processing CreateTablet for tablet 04b1317a33c34953b84b5a8c6598368f (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 03:50:41.726630 2366 tablet_service.cc:1467] Processing CreateTablet for tablet 750d734a677a4861bca565b69d29396e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260806 03:50:41.798820 2461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223d3e4f95dc4407a7abb4d09b0cceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.805202 2356 tablet_service.cc:1467] Processing CreateTablet for tablet f3ae6cf9eba9410d8307ed1da6a62ea6 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260806 03:50:41.781405 2642 raft_consensus.cc:348] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:41.807006 2642 raft_consensus.cc:374] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:41.807328 2642 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:41.764969 2363 tablet_service.cc:1467] Processing CreateTablet for tablet 6796c2cfd90d4dc9ab9759b7845f0fd0 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260806 03:50:41.809157 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 750d734a677a4861bca565b69d29396e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.809640 2473 tablet_service.cc:1467] Processing CreateTablet for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 03:50:41.811103 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.813334 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6796c2cfd90d4dc9ab9759b7845f0fd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.830615 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b1317a33c34953b84b5a8c6598368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.832212 2356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ae6cf9eba9410d8307ed1da6a62ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.814224 2359 tablet_service.cc:1467] Processing CreateTablet for tablet 223d3e4f95dc4407a7abb4d09b0cceb7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260806 03:50:41.816522 2644 tablet_bootstrap.cc:492] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:41.834620 2644 ts_tablet_manager.cc:1397] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.122s user 0.014s sys 0.007s
I20260806 03:50:41.834935 2359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 223d3e4f95dc4407a7abb4d09b0cceb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.817148 2642 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:41.827360 2566 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb2162a24b741ebbc9732b674259c2d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 03:50:41.837131 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb2162a24b741ebbc9732b674259c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.848151 2472 tablet_service.cc:1467] Processing CreateTablet for tablet b91e8fd3fd1340d2bd8db09b77bd7b0f (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 03:50:41.850445 2472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91e8fd3fd1340d2bd8db09b77bd7b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.854130 2644 raft_consensus.cc:348] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:41.855013 2644 raft_consensus.cc:374] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:41.855319 2644 raft_consensus.cc:729] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:41.858958 2643 tablet_bootstrap.cc:654] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:41.861814 2649 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:41.862366 2649 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:41.869411 2642 ts_tablet_manager.cc:1428] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.091s user 0.005s sys 0.005s
I20260806 03:50:41.870365 2644 consensus_queue.cc:260] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:41.875023 2623 tablet.cc:2369] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 03:50:41.876232 2622 heartbeater.cc:503] Master 127.0.183.62:40035 was elected leader, sending a full tablet report...
I20260806 03:50:41.889969 2358 tablet_service.cc:1467] Processing CreateTablet for tablet b91e8fd3fd1340d2bd8db09b77bd7b0f (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260806 03:50:41.891362 2466 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb2162a24b741ebbc9732b674259c2d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 03:50:41.899071 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb2162a24b741ebbc9732b674259c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.904508 2414 heartbeater.cc:503] Master 127.0.183.62:40035 was elected leader, sending a full tablet report...
I20260806 03:50:41.908360 2642 tablet_bootstrap.cc:492] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:41.913664 2364 tablet_service.cc:1467] Processing CreateTablet for tablet 3fb2162a24b741ebbc9732b674259c2d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260806 03:50:41.929029 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 04b1317a33c34953b84b5a8c6598368f (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260806 03:50:41.891997 2358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b91e8fd3fd1340d2bd8db09b77bd7b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.892793 2649 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:41.937041 2367 tablet_service.cc:1467] Processing CreateTablet for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260806 03:50:41.938597 2367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.969704 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04b1317a33c34953b84b5a8c6598368f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:41.984220 2355 tablet_service.cc:1467] Processing CreateTablet for tablet da90827d7f0646f5bb848777586aae32 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 03:50:41.986532 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90827d7f0646f5bb848777586aae32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.018532 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fb2162a24b741ebbc9732b674259c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.029744 2517 heartbeater.cc:503] Master 127.0.183.62:40035 was elected leader, sending a full tablet report...
I20260806 03:50:42.029433 2465 tablet_service.cc:1467] Processing CreateTablet for tablet 05e17a7474ab4829922b72cc624de950 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 03:50:42.031013 2465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e17a7474ab4829922b72cc624de950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.040714 2468 tablet_service.cc:1467] Processing CreateTablet for tablet da90827d7f0646f5bb848777586aae32 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 03:50:42.048409 2642 tablet_bootstrap.cc:654] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:42.053184 2464 tablet_service.cc:1467] Processing CreateTablet for tablet 1d90046b84bc4833ac8950bd501840d7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 03:50:42.045411 2467 tablet_service.cc:1467] Processing CreateTablet for tablet 97ea60c38c204c989b4ca800e8478f18 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 03:50:42.053186 2357 tablet_service.cc:1467] Processing CreateTablet for tablet 97ea60c38c204c989b4ca800e8478f18 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 03:50:42.056880 2644 ts_tablet_manager.cc:1428] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.222s user 0.013s sys 0.014s
I20260806 03:50:42.057066 2357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ea60c38c204c989b4ca800e8478f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.072300 2644 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:42.080387 2468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90827d7f0646f5bb848777586aae32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.096446 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d90046b84bc4833ac8950bd501840d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.097968 2644 tablet_bootstrap.cc:654] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:42.075994 2359 tablet_service.cc:1467] Processing CreateTablet for tablet 1d90046b84bc4833ac8950bd501840d7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 03:50:42.086808 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:42.111166 2569 tablet_service.cc:1467] Processing CreateTablet for tablet da90827d7f0646f5bb848777586aae32 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260806 03:50:42.113183 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90827d7f0646f5bb848777586aae32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.117460 2467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ea60c38c204c989b4ca800e8478f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.118891 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 05e17a7474ab4829922b72cc624de950 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 03:50:42.121196 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 97ea60c38c204c989b4ca800e8478f18 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260806 03:50:42.121672 2466 tablet_service.cc:1467] Processing CreateTablet for tablet ef53ef49663446658709042138ce12f7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 03:50:42.122612 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97ea60c38c204c989b4ca800e8478f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.122964 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef53ef49663446658709042138ce12f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.125367 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e17a7474ab4829922b72cc624de950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.125136 2566 tablet_service.cc:1467] Processing CreateTablet for tablet 1d90046b84bc4833ac8950bd501840d7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260806 03:50:42.126482 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d90046b84bc4833ac8950bd501840d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:50:42.126710 2534 leader_election.cc:341] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.079186 2363 tablet_service.cc:1467] Processing CreateTablet for tablet 05e17a7474ab4829922b72cc624de950 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260806 03:50:42.131824 2364 tablet_service.cc:1467] Processing CreateTablet for tablet ef53ef49663446658709042138ce12f7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 03:50:42.136242 2359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d90046b84bc4833ac8950bd501840d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.080648 2643 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:42.157124 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:42.157770 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05e17a7474ab4829922b72cc624de950. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.145442 2643 ts_tablet_manager.cc:1397] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.373s user 0.023s sys 0.027s
I20260806 03:50:42.163324 2643 raft_consensus.cc:348] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.164569 2643 raft_consensus.cc:374] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:42.165078 2643 raft_consensus.cc:729] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
W20260806 03:50:42.165445 2538 leader_election.cc:341] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.165968 2538 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:42.166575 2643 consensus_queue.cc:260] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.171207 2647 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:42.179617 2644 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:42.180382 2644 ts_tablet_manager.cc:1397] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.115s user 0.011s sys 0.004s
I20260806 03:50:42.183532 2644 raft_consensus.cc:348] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.184463 2644 raft_consensus.cc:374] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:42.184755 2644 raft_consensus.cc:729] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:42.185627 2644 consensus_queue.cc:260] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.188668 2644 ts_tablet_manager.cc:1428] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:50:42.189576 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef53ef49663446658709042138ce12f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.189833 2644 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:42.193467 2642 tablet_bootstrap.cc:492] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:42.194196 2642 ts_tablet_manager.cc:1397] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.311s user 0.023s sys 0.000s
I20260806 03:50:42.205201 2644 tablet_bootstrap.cc:654] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:42.206296 2367 tablet_service.cc:1467] Processing CreateTablet for tablet f1d501df3fe34842812664d2f3ec81fb (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 03:50:42.207862 2367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d501df3fe34842812664d2f3ec81fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.211057 2464 tablet_service.cc:1467] Processing CreateTablet for tablet f1d501df3fe34842812664d2f3ec81fb (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 03:50:42.212054 2570 tablet_service.cc:1467] Processing CreateTablet for tablet f1d501df3fe34842812664d2f3ec81fb (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260806 03:50:42.212504 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d501df3fe34842812664d2f3ec81fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.213455 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1d501df3fe34842812664d2f3ec81fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.220238 2568 tablet_service.cc:1467] Processing CreateTablet for tablet dfa9a0499fc04e10aefa0602a7adbbb1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 03:50:42.221707 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa9a0499fc04e10aefa0602a7adbbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.222548 2643 ts_tablet_manager.cc:1428] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.062s user 0.013s sys 0.004s
I20260806 03:50:42.223821 2643 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:42.225095 2355 tablet_service.cc:1467] Processing CreateTablet for tablet dfa9a0499fc04e10aefa0602a7adbbb1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 03:50:42.226670 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa9a0499fc04e10aefa0602a7adbbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.230849 2643 tablet_bootstrap.cc:654] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:42.235394 2561 tablet_service.cc:1467] Processing CreateTablet for tablet ef53ef49663446658709042138ce12f7 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260806 03:50:42.236893 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef53ef49663446658709042138ce12f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.236471 2466 tablet_service.cc:1467] Processing CreateTablet for tablet dfa9a0499fc04e10aefa0602a7adbbb1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260806 03:50:42.238270 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa9a0499fc04e10aefa0602a7adbbb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.247438 2359 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 03:50:42.249409 2359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.249521 2465 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 03:50:42.251005 2465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.253078 2366 tablet_service.cc:1467] Processing CreateTablet for tablet 7724acebda824fbcbf3d5bbc6a4b820c (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 03:50:42.254388 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7724acebda824fbcbf3d5bbc6a4b820c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.257139 2569 tablet_service.cc:1467] Processing CreateTablet for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260806 03:50:42.258600 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.273120 2642 raft_consensus.cc:348] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.273991 2642 raft_consensus.cc:374] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:42.274284 2642 raft_consensus.cc:729] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:42.275142 2642 consensus_queue.cc:260] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.280335 2644 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:42.280936 2644 ts_tablet_manager.cc:1397] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.091s user 0.020s sys 0.007s
I20260806 03:50:42.284795 2644 raft_consensus.cc:348] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.285336 2652 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:42.285773 2644 raft_consensus.cc:374] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:42.286141 2644 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:42.285928 2652 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.287225 2644 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.288383 2461 tablet_service.cc:1467] Processing CreateTablet for tablet 7724acebda824fbcbf3d5bbc6a4b820c (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 03:50:42.289806 2461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7724acebda824fbcbf3d5bbc6a4b820c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.294469 2652 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:42.295451 2652 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:42.295943 2652 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.297974 2652 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:42.348308 2566 tablet_service.cc:1467] Processing CreateTablet for tablet 7724acebda824fbcbf3d5bbc6a4b820c (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260806 03:50:42.349941 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7724acebda824fbcbf3d5bbc6a4b820c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.356221 2647 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:42.356966 2647 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.360590 2647 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:42.362032 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:42.363992 2538 leader_election.cc:341] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.365935 2642 ts_tablet_manager.cc:1428] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.171s user 0.009s sys 0.000s
I20260806 03:50:42.367515 2642 tablet_bootstrap.cc:492] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:42.369086 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:42.370025 2379 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:42.370313 2667 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:42.372198 2534 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:42.375692 2647 raft_consensus.cc:2781] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:42.376187 2647 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:42.377024 2647 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:42.375824 2355 tablet_service.cc:1467] Processing CreateTablet for tablet 2401ce53befd43c6a455e89c5df49a28 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 03:50:42.379369 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2401ce53befd43c6a455e89c5df49a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.381687 2464 tablet_service.cc:1467] Processing CreateTablet for tablet 2401ce53befd43c6a455e89c5df49a28 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 03:50:42.383603 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2401ce53befd43c6a455e89c5df49a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.371151 2667 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.394615 2569 tablet_service.cc:1467] Processing CreateTablet for tablet 2401ce53befd43c6a455e89c5df49a28 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260806 03:50:42.408830 2465 tablet_service.cc:1467] Processing CreateTablet for tablet 39115846d5144463b3a3a8e09e8b4500 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 03:50:42.409459 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 39115846d5144463b3a3a8e09e8b4500 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 03:50:42.410599 2465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39115846d5144463b3a3a8e09e8b4500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.418608 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2401ce53befd43c6a455e89c5df49a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.419019 2357 tablet_service.cc:1467] Processing CreateTablet for tablet 39115846d5144463b3a3a8e09e8b4500 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260806 03:50:42.424903 2466 tablet_service.cc:1467] Processing CreateTablet for tablet 0d63bf35a2e44ee881530227ba8e7e99 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 03:50:42.425880 2357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39115846d5144463b3a3a8e09e8b4500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.427433 2473 tablet_service.cc:1467] Processing CreateTablet for tablet 507574ca8b05468888fa8465ccdb2f80 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 03:50:42.419373 2467 tablet_service.cc:1467] Processing CreateTablet for tablet d48e3c6f820045128d5e9bad24b98699 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 03:50:42.420637 2366 tablet_service.cc:1467] Processing CreateTablet for tablet d48e3c6f820045128d5e9bad24b98699 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 03:50:42.430332 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48e3c6f820045128d5e9bad24b98699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.430678 2667 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:42.420909 2563 tablet_service.cc:1467] Processing CreateTablet for tablet d48e3c6f820045128d5e9bad24b98699 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260806 03:50:42.422967 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 0d63bf35a2e44ee881530227ba8e7e99 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 03:50:42.441092 2644 ts_tablet_manager.cc:1428] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.160s user 0.045s sys 0.053s
I20260806 03:50:42.440950 2363 tablet_service.cc:1467] Processing CreateTablet for tablet 0d63bf35a2e44ee881530227ba8e7e99 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260806 03:50:42.442462 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d63bf35a2e44ee881530227ba8e7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.442431 2644 tablet_bootstrap.cc:492] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:42.454166 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d63bf35a2e44ee881530227ba8e7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.455093 2467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48e3c6f820045128d5e9bad24b98699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.456437 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507574ca8b05468888fa8465ccdb2f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.469115 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 507574ca8b05468888fa8465ccdb2f80 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 03:50:42.471874 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d48e3c6f820045128d5e9bad24b98699. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.473480 2567 tablet_service.cc:1467] Processing CreateTablet for tablet a930c23072484f7ea5974173f5843bfe (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 03:50:42.474833 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a930c23072484f7ea5974173f5843bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.476395 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39115846d5144463b3a3a8e09e8b4500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.478170 2468 tablet_service.cc:1467] Processing CreateTablet for tablet a930c23072484f7ea5974173f5843bfe (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 03:50:42.479645 2468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a930c23072484f7ea5974173f5843bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.480330 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d63bf35a2e44ee881530227ba8e7e99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.486997 2364 tablet_service.cc:1467] Processing CreateTablet for tablet 507574ca8b05468888fa8465ccdb2f80 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260806 03:50:42.487685 2359 tablet_service.cc:1467] Processing CreateTablet for tablet a930c23072484f7ea5974173f5843bfe (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260806 03:50:42.488463 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507574ca8b05468888fa8465ccdb2f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.489172 2359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a930c23072484f7ea5974173f5843bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.490689 2571 tablet_service.cc:1467] Processing CreateTablet for tablet fb91ba465b9845129828f07c4d452970 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 03:50:42.493098 2463 tablet_service.cc:1467] Processing CreateTablet for tablet fb91ba465b9845129828f07c4d452970 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 03:50:42.494410 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb91ba465b9845129828f07c4d452970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.495316 2469 tablet_service.cc:1467] Processing CreateTablet for tablet 78ff87b2196e4600aed5d8442f9b9bdd (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 03:50:42.496217 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb91ba465b9845129828f07c4d452970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.496644 2469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ff87b2196e4600aed5d8442f9b9bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.498036 2358 tablet_service.cc:1467] Processing CreateTablet for tablet 78ff87b2196e4600aed5d8442f9b9bdd (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 03:50:42.499334 2358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ff87b2196e4600aed5d8442f9b9bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.501088 2356 tablet_service.cc:1467] Processing CreateTablet for tablet fb91ba465b9845129828f07c4d452970 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260806 03:50:42.502365 2356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb91ba465b9845129828f07c4d452970. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.505445 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 78ff87b2196e4600aed5d8442f9b9bdd (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260806 03:50:42.506824 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ff87b2196e4600aed5d8442f9b9bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.509105 2472 tablet_service.cc:1467] Processing CreateTablet for tablet 4fed83d97aac42e4b3e69249098b6111 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 03:50:42.510481 2472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fed83d97aac42e4b3e69249098b6111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.510234 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 4fed83d97aac42e4b3e69249098b6111 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 03:50:42.510389 2647 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.511557 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fed83d97aac42e4b3e69249098b6111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.542346 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507574ca8b05468888fa8465ccdb2f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.543514 2642 tablet_bootstrap.cc:654] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:42.543773 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:42.544684 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:42.545519 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:42.546242 2331 leader_election.cc:341] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.546435 2583 raft_consensus.cc:2379] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:50:42.547901 2331 leader_election.cc:341] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.549335 2567 tablet_service.cc:1467] Processing CreateTablet for tablet 4fed83d97aac42e4b3e69249098b6111 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260806 03:50:42.553580 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:42.554330 2379 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:42.562585 2379 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:42.562743 2365 tablet_service.cc:1467] Processing CreateTablet for tablet 01b082125feb462d9ba064ed2acf4dfd (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 03:50:42.490675 2643 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:42.564446 2534 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:42.564664 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b082125feb462d9ba064ed2acf4dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.564729 2643 ts_tablet_manager.cc:1397] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.341s user 0.028s sys 0.047s
I20260806 03:50:42.576244 2647 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:42.595901 2661 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:42.597128 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fed83d97aac42e4b3e69249098b6111. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.617147 2647 raft_consensus.cc:2781] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:42.617744 2647 raft_consensus.cc:686] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:42.618633 2647 consensus_queue.cc:237] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.622059 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 01b082125feb462d9ba064ed2acf4dfd (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 03:50:42.623536 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b082125feb462d9ba064ed2acf4dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.625777 2566 tablet_service.cc:1467] Processing CreateTablet for tablet 84da474ba5944a7a84d7c76403b92a63 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 03:50:42.627170 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84da474ba5944a7a84d7c76403b92a63. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:50:42.632218 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 67916.7us
I20260806 03:50:42.606603 2644 tablet_bootstrap.cc:654] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:42.598304 2643 raft_consensus.cc:348] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.613293 2661 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.637156 2643 raft_consensus.cc:374] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:42.639297 2643 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:42.640286 2643 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.641942 2355 tablet_service.cc:1467] Processing CreateTablet for tablet 2ccb469db1664d3e941e7a360d048af2 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 03:50:42.643615 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ccb469db1664d3e941e7a360d048af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.665863 2564 tablet_service.cc:1467] Processing CreateTablet for tablet 32d01287577744949bfebb63c59f3db6 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 03:50:42.667595 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d01287577744949bfebb63c59f3db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.677023 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:42.679067 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:42.652045 2661 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:42.655179 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:42.697373 2461 tablet_service.cc:1467] Processing CreateTablet for tablet 01b082125feb462d9ba064ed2acf4dfd (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260806 03:50:42.625793 2367 tablet_service.cc:1467] Processing CreateTablet for tablet 84da474ba5944a7a84d7c76403b92a63 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 03:50:42.707890 2367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84da474ba5944a7a84d7c76403b92a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.699249 2356 tablet_service.cc:1467] Processing CreateTablet for tablet 32d01287577744949bfebb63c59f3db6 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 03:50:42.702999 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:42.703706 2484 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:42.706164 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 2ccb469db1664d3e941e7a360d048af2 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
W20260806 03:50:42.723625 2332 leader_election.cc:341] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.724633 2332 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:42.729813 2472 tablet_service.cc:1467] Processing CreateTablet for tablet 84da474ba5944a7a84d7c76403b92a63 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260806 03:50:42.731544 2667 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:42.734894 2469 tablet_service.cc:1467] Processing CreateTablet for tablet 2ccb469db1664d3e941e7a360d048af2 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260806 03:50:42.737660 2461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b082125feb462d9ba064ed2acf4dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.746847 2472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84da474ba5944a7a84d7c76403b92a63. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:50:42.740430 2538 leader_election.cc:341] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.741101 2473 tablet_service.cc:1467] Processing CreateTablet for tablet 32d01287577744949bfebb63c59f3db6 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260806 03:50:42.750334 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d01287577744949bfebb63c59f3db6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:50:42.740476 2332 leader_election.cc:341] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.760702 2332 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:42.761579 2667 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:42.762985 2332 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:42.763819 2667 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:42.764225 2667 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:42.764544 2667 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:42.770521 2463 tablet_service.cc:1467] Processing CreateTablet for tablet 5489408bdcc042f3919eae139ea82e7a (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 03:50:42.774385 2356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32d01287577744949bfebb63c59f3db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.775254 2643 ts_tablet_manager.cc:1428] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.190s user 0.016s sys 0.033s
W20260806 03:50:42.703020 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:42.767452 2359 tablet_service.cc:1467] Processing CreateTablet for tablet 5489408bdcc042f3919eae139ea82e7a (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 03:50:42.784749 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ccb469db1664d3e941e7a360d048af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.789760 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5489408bdcc042f3919eae139ea82e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.833994 2357 tablet_service.cc:1467] Processing CreateTablet for tablet f5efa489aaa94f9988b77e5efc032e5e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 03:50:42.835528 2357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5efa489aaa94f9988b77e5efc032e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.847203 2465 tablet_service.cc:1467] Processing CreateTablet for tablet 7901535e3d4042389b2a8853a881ff1d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 03:50:42.807070 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:42.810102 2363 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2fa42f901d4b37bbfd3baccb897807 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 03:50:42.813782 2661 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:42.816527 2563 tablet_service.cc:1467] Processing CreateTablet for tablet 5489408bdcc042f3919eae139ea82e7a (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260806 03:50:42.878859 2365 tablet_service.cc:1467] Processing CreateTablet for tablet f327872ee26848bbaf04c03cc38edd54 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
W20260806 03:50:42.881275 2436 leader_election.cc:341] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.882951 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5489408bdcc042f3919eae139ea82e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.849757 2359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5489408bdcc042f3919eae139ea82e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.893695 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2fa42f901d4b37bbfd3baccb897807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.850848 2667 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.896363 2667 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:42.867888 2364 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb0e9b9f35b4cd488afa1ae281597bc (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 03:50:42.897967 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb0e9b9f35b4cd488afa1ae281597bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.898367 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:42.899791 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:42.900569 2481 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:42.854615 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:42.857735 2467 tablet_service.cc:1467] Processing CreateTablet for tablet f5efa489aaa94f9988b77e5efc032e5e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260806 03:50:42.904578 2379 raft_consensus.cc:2379] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:50:42.863215 2218 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 59534.8us
I20260806 03:50:42.864075 2469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ccb469db1664d3e941e7a360d048af2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.848183 2568 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2fa42f901d4b37bbfd3baccb897807 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 03:50:42.908726 2365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f327872ee26848bbaf04c03cc38edd54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.796301 2366 tablet_service.cc:1467] Processing CreateTablet for tablet 27aa6efa6253470d9fb619f5536e8795 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 03:50:42.878800 2661 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.872895 2642 tablet_bootstrap.cc:492] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:42.910818 2661 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:42.911301 2642 ts_tablet_manager.cc:1397] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.544s user 0.011s sys 0.018s
I20260806 03:50:42.864400 2569 tablet_service.cc:1467] Processing CreateTablet for tablet 27aa6efa6253470d9fb619f5536e8795 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260806 03:50:42.859177 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 7901535e3d4042389b2a8853a881ff1d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 03:50:42.922376 2467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5efa489aaa94f9988b77e5efc032e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.908806 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2fa42f901d4b37bbfd3baccb897807. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:50:42.899992 2331 leader_election.cc:341] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.851169 2466 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb0e9b9f35b4cd488afa1ae281597bc (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 03:50:42.917660 2429 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:42.919869 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27aa6efa6253470d9fb619f5536e8795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.920646 2468 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2fa42f901d4b37bbfd3baccb897807 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260806 03:50:42.949537 2468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2fa42f901d4b37bbfd3baccb897807. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.943965 2642 raft_consensus.cc:348] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.955608 2642 raft_consensus.cc:374] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:42.955964 2642 raft_consensus.cc:729] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:42.957012 2642 consensus_queue.cc:260] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.957212 2236 catalog_manager.cc:5495] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:42.959632 2642 ts_tablet_manager.cc:1428] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.048s user 0.005s sys 0.003s
I20260806 03:50:42.963063 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7901535e3d4042389b2a8853a881ff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.963559 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb0e9b9f35b4cd488afa1ae281597bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.964561 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27aa6efa6253470d9fb619f5536e8795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:42.977186 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:42.975174 2473 tablet_service.cc:1467] Processing CreateTablet for tablet f327872ee26848bbaf04c03cc38edd54 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 03:50:42.781049 2464 tablet_service.cc:1467] Processing CreateTablet for tablet 27aa6efa6253470d9fb619f5536e8795 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
W20260806 03:50:42.981419 2436 leader_election.cc:341] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:42.982437 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:42.816527 2567 tablet_service.cc:1467] Processing CreateTablet for tablet 0cb0e9b9f35b4cd488afa1ae281597bc (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260806 03:50:42.989055 2644 tablet_bootstrap.cc:492] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:42.989471 2642 tablet_bootstrap.cc:492] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:42.989645 2644 ts_tablet_manager.cc:1397] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.547s user 0.065s sys 0.089s
I20260806 03:50:42.993366 2644 raft_consensus.cc:348] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:42.994221 2644 raft_consensus.cc:374] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:42.994524 2644 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:42.995214 2644 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.019873 2465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7901535e3d4042389b2a8853a881ff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.002416 2358 tablet_service.cc:1467] Processing CreateTablet for tablet 7901535e3d4042389b2a8853a881ff1d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260806 03:50:42.975711 2643 tablet_bootstrap.cc:492] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:43.060809 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cb0e9b9f35b4cd488afa1ae281597bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.067005 2570 tablet_service.cc:1467] Processing CreateTablet for tablet f5efa489aaa94f9988b77e5efc032e5e (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
W20260806 03:50:43.073645 2429 leader_election.cc:341] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.074211 2429 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:43.075112 2363 tablet_service.cc:1467] Processing CreateTablet for tablet 6364534d89cb43079c8dafda6b5b8875 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 03:50:43.075960 2674 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:43.074209 2358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7901535e3d4042389b2a8853a881ff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.081221 2644 ts_tablet_manager.cc:1428] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.091s user 0.008s sys 0.000s
I20260806 03:50:43.088105 2647 consensus_queue.cc:1035] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:43.093648 2356 tablet_service.cc:1467] Processing CreateTablet for tablet 8db383c38ecd45f2b2cb8c46ff360638 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 03:50:43.095297 2356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db383c38ecd45f2b2cb8c46ff360638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.095831 2472 tablet_service.cc:1467] Processing CreateTablet for tablet 8db383c38ecd45f2b2cb8c46ff360638 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 03:50:43.098866 2461 tablet_service.cc:1467] Processing CreateTablet for tablet c4f2c284701f44e495902166ab19b186 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 03:50:43.104496 2566 tablet_service.cc:1467] Processing CreateTablet for tablet 8db383c38ecd45f2b2cb8c46ff360638 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260806 03:50:43.082993 2571 tablet_service.cc:1467] Processing CreateTablet for tablet f327872ee26848bbaf04c03cc38edd54 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260806 03:50:43.105952 2364 tablet_service.cc:1467] Processing CreateTablet for tablet 95f6f3de31d442cfa812406b89ee2fe3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 03:50:43.088152 2644 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.086534 2642 tablet_bootstrap.cc:654] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.071699 2481 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:43.105940 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db383c38ecd45f2b2cb8c46ff360638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.095830 2367 tablet_service.cc:1467] Processing CreateTablet for tablet c4f2c284701f44e495902166ab19b186 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260806 03:50:43.120851 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5efa489aaa94f9988b77e5efc032e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.116463 2661 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:43.126140 2571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f327872ee26848bbaf04c03cc38edd54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.107070 2561 tablet_service.cc:1467] Processing CreateTablet for tablet 6364534d89cb43079c8dafda6b5b8875 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 03:50:43.132925 2667 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.133299 2363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6364534d89cb43079c8dafda6b5b8875. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.133993 2367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f2c284701f44e495902166ab19b186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.133563 2667 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.136050 2667 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.137074 2364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f6f3de31d442cfa812406b89ee2fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.137439 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:43.138164 2583 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:43.118144 2461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f2c284701f44e495902166ab19b186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.139227 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.142256 2472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db383c38ecd45f2b2cb8c46ff360638. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.143065 2473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f327872ee26848bbaf04c03cc38edd54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.141345 2331 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:43.147157 2667 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:43.147584 2667 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:43.147931 2667 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:43.152159 2332 leader_election.cc:341] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.158701 2643 tablet_bootstrap.cc:654] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.159523 2464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27aa6efa6253470d9fb619f5536e8795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.162910 2332 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:43.099864 2355 tablet_service.cc:1467] Processing CreateTablet for tablet d2ddf38aaea54266a0a9e6100c48fc0d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 03:50:43.166196 2647 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.173298 2647 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.175904 2561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6364534d89cb43079c8dafda6b5b8875. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.176738 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:43.175853 2647 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:50:43.178347 2534 leader_election.cc:341] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.178541 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.169445 2564 tablet_service.cc:1467] Processing CreateTablet for tablet c4f2c284701f44e495902166ab19b186 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
W20260806 03:50:43.188364 2538 leader_election.cc:341] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.189059 2538 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:43.189105 2463 tablet_service.cc:1467] Processing CreateTablet for tablet 6364534d89cb43079c8dafda6b5b8875 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260806 03:50:43.191071 2564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4f2c284701f44e495902166ab19b186. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.192369 2562 tablet_service.cc:1467] Processing CreateTablet for tablet d2ddf38aaea54266a0a9e6100c48fc0d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 03:50:43.193717 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ddf38aaea54266a0a9e6100c48fc0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.200006 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ddf38aaea54266a0a9e6100c48fc0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.222939 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 95f6f3de31d442cfa812406b89ee2fe3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 03:50:43.225550 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f6f3de31d442cfa812406b89ee2fe3. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:50:43.212360 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 49555.9us
I20260806 03:50:43.212270 2690 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:43.226166 2463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6364534d89cb43079c8dafda6b5b8875. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.254531 2644 tablet_bootstrap.cc:654] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.274930 2642 tablet_bootstrap.cc:492] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:43.275545 2642 ts_tablet_manager.cc:1397] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.286s user 0.005s sys 0.015s
I20260806 03:50:43.267407 2688 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:43.212903 2667 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.283114 2357 tablet_service.cc:1467] Processing CreateTablet for tablet ead4b99d10d24d13ba7f07dee24f79e3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 03:50:43.284487 2469 tablet_service.cc:1467] Processing CreateTablet for tablet 95f6f3de31d442cfa812406b89ee2fe3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260806 03:50:43.293365 2357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead4b99d10d24d13ba7f07dee24f79e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.220515 2467 tablet_service.cc:1467] Processing CreateTablet for tablet d2ddf38aaea54266a0a9e6100c48fc0d (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260806 03:50:43.298203 2569 tablet_service.cc:1467] Processing CreateTablet for tablet 940c72b441104eabbcb3ea23683268c1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 03:50:43.299633 2569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c72b441104eabbcb3ea23683268c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.266307 2366 tablet_service.cc:1467] Processing CreateTablet for tablet 940c72b441104eabbcb3ea23683268c1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260806 03:50:43.300546 2563 tablet_service.cc:1467] Processing CreateTablet for tablet 8655b80017d7422082efde215e797b5b (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 03:50:43.301239 2366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c72b441104eabbcb3ea23683268c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.302992 2642 raft_consensus.cc:348] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.303790 2642 raft_consensus.cc:374] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.313309 2567 tablet_service.cc:1467] Processing CreateTablet for tablet ead4b99d10d24d13ba7f07dee24f79e3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 03:50:43.313832 2355 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 03:50:43.317466 2356 tablet_service.cc:1467] Processing CreateTablet for tablet 8655b80017d7422082efde215e797b5b (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 03:50:43.325461 2667 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.328012 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:50:43.338346 2218 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 45866.2us
I20260806 03:50:43.340183 2583 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.340840 2469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f6f3de31d442cfa812406b89ee2fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.331216 2642 raft_consensus.cc:729] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.348840 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8655b80017d7422082efde215e797b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.358613 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:43.356276 2467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2ddf38aaea54266a0a9e6100c48fc0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.359293 2642 consensus_queue.cc:260] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.357628 2468 tablet_service.cc:1467] Processing CreateTablet for tablet 940c72b441104eabbcb3ea23683268c1 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
W20260806 03:50:43.364390 2332 leader_election.cc:341] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.396140 2643 tablet_bootstrap.cc:492] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:43.396889 2643 ts_tablet_manager.cc:1397] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.463s user 0.046s sys 0.030s
W20260806 03:50:43.366344 2538 consensus_peers.cc:487] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:43.376511 2468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c72b441104eabbcb3ea23683268c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.384380 2567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead4b99d10d24d13ba7f07dee24f79e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.410739 2568 tablet_service.cc:1467] Processing CreateTablet for tablet d7a939d2320d4a478b95733e1579e2b3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 03:50:43.362112 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 03:50:43.418336 2583 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:43.377491 2355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.432047 2331 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:43.435667 2667 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:43.370213 2466 tablet_service.cc:1467] Processing CreateTablet for tablet ead4b99d10d24d13ba7f07dee24f79e3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260806 03:50:43.384864 2642 ts_tablet_manager.cc:1428] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.109s user 0.006s sys 0.005s
I20260806 03:50:43.440274 2667 raft_consensus.cc:686] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:43.438508 2690 raft_consensus.cc:686] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:43.443562 2568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7a939d2320d4a478b95733e1579e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.443753 2356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8655b80017d7422082efde215e797b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.449040 2466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead4b99d10d24d13ba7f07dee24f79e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.452720 2461 tablet_service.cc:1467] Processing CreateTablet for tablet d7a939d2320d4a478b95733e1579e2b3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 03:50:43.452422 2667 consensus_queue.cc:237] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.401072 2643 raft_consensus.cc:348] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.405944 2472 tablet_service.cc:1467] Processing CreateTablet for tablet 8655b80017d7422082efde215e797b5b (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260806 03:50:43.481303 2461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7a939d2320d4a478b95733e1579e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.483076 2465 tablet_service.cc:1467] Processing CreateTablet for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260806 03:50:43.484094 2472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8655b80017d7422082efde215e797b5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.484165 2643 raft_consensus.cc:374] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.484702 2643 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.485204 2642 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:43.486486 2643 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.489393 2643 ts_tablet_manager.cc:1428] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.092s user 0.009s sys 0.001s
I20260806 03:50:43.491957 2643 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:43.391295 2359 tablet_service.cc:1467] Processing CreateTablet for tablet d7a939d2320d4a478b95733e1579e2b3 (DEFAULT_TABLE table=test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260806 03:50:43.495536 2690 consensus_queue.cc:237] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.508164 2359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7a939d2320d4a478b95733e1579e2b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.502776 2642 tablet_bootstrap.cc:654] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.511365 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.523845 2644 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:43.524296 2465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:50:43.524490 2644 ts_tablet_manager.cc:1397] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.437s user 0.029s sys 0.006s
I20260806 03:50:43.528103 2661 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.528621 2661 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.530817 2661 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.532590 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:43.535270 2643 tablet_bootstrap.cc:654] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.534996 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:43.536131 2436 leader_election.cc:341] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:43.536716 2429 leader_election.cc:341] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.537191 2429 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:43.539376 2661 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:43.527680 2644 raft_consensus.cc:348] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.543877 2644 raft_consensus.cc:374] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.554840 2644 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.568864 2644 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.569680 2236 catalog_manager.cc:5495] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:43.571604 2644 ts_tablet_manager.cc:1428] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.047s user 0.003s sys 0.002s
I20260806 03:50:43.572000 2236 catalog_manager.cc:5495] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:43.573850 2642 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:43.574405 2642 ts_tablet_manager.cc:1397] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.011s
I20260806 03:50:43.583369 2642 raft_consensus.cc:348] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.588886 2642 raft_consensus.cc:374] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.589247 2642 raft_consensus.cc:729] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.591596 2644 tablet_bootstrap.cc:492] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.591207 2642 consensus_queue.cc:260] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.595395 2642 ts_tablet_manager.cc:1428] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260806 03:50:43.596521 2642 tablet_bootstrap.cc:492] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:43.598475 2644 tablet_bootstrap.cc:654] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.603240 2642 tablet_bootstrap.cc:654] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.604297 2643 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:43.604861 2643 ts_tablet_manager.cc:1397] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.113s user 0.015s sys 0.000s
I20260806 03:50:43.608069 2643 raft_consensus.cc:348] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.608882 2643 raft_consensus.cc:374] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.609176 2643 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.610044 2643 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.616799 2643 ts_tablet_manager.cc:1428] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260806 03:50:43.617704 2643 tablet_bootstrap.cc:492] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:43.620828 2642 tablet_bootstrap.cc:492] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:43.621408 2642 ts_tablet_manager.cc:1397] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.008s
I20260806 03:50:43.625787 2643 tablet_bootstrap.cc:654] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.626237 2642 raft_consensus.cc:348] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.627269 2642 raft_consensus.cc:374] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.627315 2644 tablet_bootstrap.cc:492] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:43.627643 2642 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.627988 2644 ts_tablet_manager.cc:1397] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.037s user 0.016s sys 0.000s
I20260806 03:50:43.628620 2642 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.632249 2644 raft_consensus.cc:348] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.633571 2642 ts_tablet_manager.cc:1428] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.012s user 0.005s sys 0.003s
I20260806 03:50:43.636313 2644 raft_consensus.cc:374] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.636806 2644 raft_consensus.cc:729] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.637411 2642 tablet_bootstrap.cc:492] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:43.637916 2644 consensus_queue.cc:260] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.642874 2644 ts_tablet_manager.cc:1428] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.014s user 0.005s sys 0.004s
I20260806 03:50:43.644841 2644 tablet_bootstrap.cc:492] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.644996 2691 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.645570 2691 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.647311 2643 tablet_bootstrap.cc:492] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:43.647888 2643 ts_tablet_manager.cc:1397] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.030s user 0.014s sys 0.012s
I20260806 03:50:43.649955 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.650213 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:43.651149 2643 raft_consensus.cc:348] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.652094 2643 raft_consensus.cc:374] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.652280 2691 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:43.652611 2643 raft_consensus.cc:729] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
W20260806 03:50:43.652016 2534 leader_election.cc:341] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:43.653409 2538 leader_election.cc:341] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.653409 2643 consensus_queue.cc:260] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.654458 2644 tablet_bootstrap.cc:654] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.654762 2538 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:43.656454 2643 ts_tablet_manager.cc:1428] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 03:50:43.656344 2691 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:43.660452 2642 tablet_bootstrap.cc:654] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.662396 2643 tablet_bootstrap.cc:492] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:43.667182 2644 tablet_bootstrap.cc:492] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:43.667834 2644 ts_tablet_manager.cc:1397] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.001s
I20260806 03:50:43.670284 2643 tablet_bootstrap.cc:654] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.670881 2644 raft_consensus.cc:348] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.671777 2644 raft_consensus.cc:374] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.672230 2644 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.673117 2644 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.680550 2644 ts_tablet_manager.cc:1428] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.012s user 0.003s sys 0.004s
I20260806 03:50:43.681620 2642 tablet_bootstrap.cc:492] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:43.682057 2644 tablet_bootstrap.cc:492] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.682193 2642 ts_tablet_manager.cc:1397] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.045s user 0.014s sys 0.009s
I20260806 03:50:43.685487 2642 raft_consensus.cc:348] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.686295 2642 raft_consensus.cc:374] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.686596 2642 raft_consensus.cc:729] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.687404 2642 consensus_queue.cc:260] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.689978 2642 ts_tablet_manager.cc:1428] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:50:43.691056 2642 tablet_bootstrap.cc:492] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:43.693688 2644 tablet_bootstrap.cc:654] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.697880 2642 tablet_bootstrap.cc:654] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.703473 2643 tablet_bootstrap.cc:492] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:43.703671 2644 tablet_bootstrap.cc:492] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:43.704142 2643 ts_tablet_manager.cc:1397] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.042s user 0.038s sys 0.000s
I20260806 03:50:43.704443 2644 ts_tablet_manager.cc:1397] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.001s
I20260806 03:50:43.707600 2644 raft_consensus.cc:348] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.707739 2643 raft_consensus.cc:348] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.708642 2644 raft_consensus.cc:374] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.708642 2643 raft_consensus.cc:374] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.709107 2644 raft_consensus.cc:729] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.709156 2643 raft_consensus.cc:729] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.710047 2644 consensus_queue.cc:260] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.715209 2643 consensus_queue.cc:260] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.716383 2642 tablet_bootstrap.cc:492] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:43.716966 2642 ts_tablet_manager.cc:1397] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.010s
I20260806 03:50:43.717218 2644 ts_tablet_manager.cc:1428] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260806 03:50:43.719086 2643 ts_tablet_manager.cc:1428] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.015s user 0.000s sys 0.010s
I20260806 03:50:43.720248 2643 tablet_bootstrap.cc:492] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:43.720683 2644 tablet_bootstrap.cc:492] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.724756 2690 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.725343 2690 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.727208 2644 tablet_bootstrap.cc:654] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.727754 2690 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.727226 2643 tablet_bootstrap.cc:654] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.729668 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.729743 2642 raft_consensus.cc:348] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.730526 2481 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:43.730670 2642 raft_consensus.cc:374] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.730701 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:43.731164 2642 raft_consensus.cc:729] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.731508 2583 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:43.731876 2332 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:43.732100 2642 consensus_queue.cc:260] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.732806 2690 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:43.733215 2690 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:43.733552 2690 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.734330 2642 ts_tablet_manager.cc:1428] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260806 03:50:43.738593 2642 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:43.744596 2690 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.747902 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:43.748183 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:43.748636 2481 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.748777 2583 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.749697 2691 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.750212 2691 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.752218 2691 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:43.754340 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:43.756672 2642 tablet_bootstrap.cc:654] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.756889 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.757278 2583 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:43.758531 2331 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:43.759536 2667 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:43.760046 2667 raft_consensus.cc:686] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:43.760941 2667 consensus_queue.cc:237] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.763281 2698 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:50:43.758672 2534 leader_election.cc:341] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:43.765378 2538 leader_election.cc:341] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.766556 2538 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:43.768208 2691 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:43.763957 2698 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.770572 2481 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:43.771308 2698 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.772192 2690 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.772269 2643 tablet_bootstrap.cc:492] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:43.773166 2643 ts_tablet_manager.cc:1397] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.053s user 0.019s sys 0.008s
I20260806 03:50:43.773572 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.775758 2644 tablet_bootstrap.cc:492] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:43.776423 2644 ts_tablet_manager.cc:1397] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.056s user 0.019s sys 0.000s
W20260806 03:50:43.777737 2332 leader_election.cc:341] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.776491 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:43.780314 2644 raft_consensus.cc:348] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.781353 2644 raft_consensus.cc:374] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.781704 2644 raft_consensus.cc:729] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
W20260806 03:50:43.782214 2331 leader_election.cc:341] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.782843 2331 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:43.782554 2644 consensus_queue.cc:260] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.784740 2690 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:43.787326 2643 raft_consensus.cc:348] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.788414 2643 raft_consensus.cc:374] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.788748 2643 raft_consensus.cc:729] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.791137 2644 ts_tablet_manager.cc:1428] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
W20260806 03:50:43.791690 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:43.792945 2644 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.801182 2644 tablet_bootstrap.cc:654] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.791755 2643 consensus_queue.cc:260] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.804138 2233 catalog_manager.cc:5495] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:43.808187 2690 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.808960 2690 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.813169 2690 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.807965 2643 ts_tablet_manager.cc:1428] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.024s user 0.005s sys 0.002s
I20260806 03:50:43.815887 2643 tablet_bootstrap.cc:492] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:43.817193 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.817950 2481 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:43.819223 2332 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:43.820142 2690 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:43.820505 2690 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:43.820854 2690 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.824337 2643 tablet_bootstrap.cc:654] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.829689 2644 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:43.830210 2644 ts_tablet_manager.cc:1397] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.003s
I20260806 03:50:43.833031 2644 raft_consensus.cc:348] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.833810 2644 raft_consensus.cc:374] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.834096 2644 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.834786 2644 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.838020 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:43.839861 2331 leader_election.cc:341] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.843578 2643 tablet_bootstrap.cc:492] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:43.844326 2644 ts_tablet_manager.cc:1428] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.014s user 0.009s sys 0.000s
I20260806 03:50:43.844338 2643 ts_tablet_manager.cc:1397] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.010s
I20260806 03:50:43.844424 2690 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.845558 2644 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.852247 2644 tablet_bootstrap.cc:654] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.853287 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:43.854079 2481 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.864446 2644 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:43.865113 2644 ts_tablet_manager.cc:1397] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20260806 03:50:43.865844 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:50:43.867340 2331 leader_election.cc:341] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.868372 2690 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.868117 2644 raft_consensus.cc:348] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.868932 2644 raft_consensus.cc:374] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.869246 2644 raft_consensus.cc:729] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.869982 2644 consensus_queue.cc:260] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.870571 2481 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:43.871954 2642 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:43.872002 2332 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:43.872504 2644 ts_tablet_manager.cc:1428] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 03:50:43.872802 2642 ts_tablet_manager.cc:1397] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.134s user 0.057s sys 0.048s
I20260806 03:50:43.873414 2690 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:43.873641 2644 tablet_bootstrap.cc:492] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.874166 2690 raft_consensus.cc:686] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:43.875047 2690 consensus_queue.cc:237] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.870474 2643 raft_consensus.cc:348] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.876371 2643 raft_consensus.cc:374] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.876731 2643 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.876955 2642 raft_consensus.cc:348] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.878019 2642 raft_consensus.cc:374] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.877862 2643 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.878383 2642 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.879384 2642 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.882789 2642 ts_tablet_manager.cc:1428] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260806 03:50:43.884307 2642 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:43.880656 2643 ts_tablet_manager.cc:1428] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.036s user 0.007s sys 0.001s
I20260806 03:50:43.891250 2642 tablet_bootstrap.cc:654] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.897303 2481 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:43.899852 2331 consensus_peers.cc:487] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 -> Peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Couldn't send request to peer fe3d17f4ad984b9088a268d58d87469c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:43.905411 2644 tablet_bootstrap.cc:654] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.911373 2643 tablet_bootstrap.cc:492] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:43.924312 2643 tablet_bootstrap.cc:654] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.926062 2691 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.926765 2691 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.928269 2644 tablet_bootstrap.cc:492] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:43.929008 2644 ts_tablet_manager.cc:1397] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.056s user 0.019s sys 0.008s
I20260806 03:50:43.929483 2691 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:43.928308 2642 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:43.930635 2642 ts_tablet_manager.cc:1397] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.047s user 0.028s sys 0.012s
I20260806 03:50:43.930711 2690 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:43.933152 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.933391 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:43.934087 2379 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
W20260806 03:50:43.934902 2538 leader_election.cc:341] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.935431 2644 raft_consensus.cc:348] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.936406 2644 raft_consensus.cc:374] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.936730 2644 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.937597 2644 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.940945 2727 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.941689 2727 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.938848 2534 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:43.944288 2644 ts_tablet_manager.cc:1428] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260806 03:50:43.943895 2233 catalog_manager.cc:5495] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:43.945292 2691 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:43.945385 2644 tablet_bootstrap.cc:492] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:43.945742 2691 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:43.946180 2691 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.946089 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.946971 2481 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:43.948284 2332 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:43.949645 2690 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:43.950044 2690 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:43.950364 2690 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.951473 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:43.952472 2583 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:43.954730 2727 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.959009 2661 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.959534 2661 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.961455 2642 raft_consensus.cc:348] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.963025 2642 raft_consensus.cc:374] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:43.963373 2642 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:43.963932 2661 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.964457 2642 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.965538 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:43.966859 2583 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:43.968284 2436 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:43.968595 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:43.969444 2379 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:43.969580 2661 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:43.970067 2661 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:43.970429 2661 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.973948 2644 tablet_bootstrap.cc:654] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:43.974952 2727 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:43.976852 2727 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.980532 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:43.980909 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:43.981654 2583 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:43.981616 2690 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:43.982030 2332 leader_election.cc:341] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:43.981973 2661 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.983417 2727 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.983922 2331 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:43.985733 2642 ts_tablet_manager.cc:1428] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.055s user 0.006s sys 0.004s
I20260806 03:50:43.986747 2642 tablet_bootstrap.cc:492] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:43.986655 2727 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:43.989099 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:43.989729 2481 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.987339 2727 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:43.991606 2661 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:43.991722 2727 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.992765 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:43.993369 2379 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:43.990238 2691 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:43.998106 2643 tablet_bootstrap.cc:492] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:43.998648 2643 ts_tablet_manager.cc:1397] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.112s user 0.012s sys 0.007s
I20260806 03:50:43.998777 2691 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:43.999665 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:44.000353 2377 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.001245 2690 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.002144 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:50:44.003516 2538 leader_election.cc:341] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.006309 2643 raft_consensus.cc:348] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.007122 2643 raft_consensus.cc:374] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.007071 2727 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.007516 2643 raft_consensus.cc:729] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.007464 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:44.008579 2581 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.008423 2643 consensus_queue.cc:260] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.009783 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:44.011466 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:44.010664 2583 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.012513 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:44.013847 2584 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.013882 2643 ts_tablet_manager.cc:1428] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.015s user 0.004s sys 0.004s
W20260806 03:50:44.017606 2332 leader_election.cc:341] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.021939 2642 tablet_bootstrap.cc:654] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.010645 2727 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.025537 2581 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:44.030783 2331 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:44.031989 2698 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.034704 2698 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:44.035323 2698 raft_consensus.cc:686] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:44.041671 2644 tablet_bootstrap.cc:492] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:44.042430 2644 ts_tablet_manager.cc:1397] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.097s user 0.012s sys 0.009s
I20260806 03:50:44.037209 2698 consensus_queue.cc:237] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.055183 2644 raft_consensus.cc:348] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.056286 2644 raft_consensus.cc:374] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.056653 2644 raft_consensus.cc:729] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.057672 2644 consensus_queue.cc:260] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.061435 2643 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:44.065951 2377 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:44.067963 2534 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:44.068418 2727 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.068912 2691 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:44.069496 2691 raft_consensus.cc:686] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:44.071130 2691 consensus_queue.cc:237] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.076186 2644 ts_tablet_manager.cc:1428] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.033s user 0.011s sys 0.000s
I20260806 03:50:44.077538 2644 tablet_bootstrap.cc:492] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:44.068907 2727 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.079820 2727 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.089776 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:44.104650 2481 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:44.105350 2661 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.111567 2661 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.114311 2661 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:44.104049 2331 leader_election.cc:341] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.109632 2236 catalog_manager.cc:5495] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:44.086971 2379 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:44.140549 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:44.142073 2483 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:44.145795 2643 tablet_bootstrap.cc:654] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.147094 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:44.157752 2727 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.157896 2429 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:44.158440 2727 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.158977 2661 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:44.165313 2642 tablet_bootstrap.cc:492] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:44.166090 2642 ts_tablet_manager.cc:1397] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.180s user 0.015s sys 0.006s
I20260806 03:50:44.167059 2584 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:44.148000 2583 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:44.169219 2698 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.175503 2727 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.175165 2698 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.175331 2331 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:44.177659 2727 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:44.178162 2727 raft_consensus.cc:686] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:44.178222 2698 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.172310 2661 raft_consensus.cc:686] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:44.179140 2727 consensus_queue.cc:237] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.181035 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:44.185567 2332 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:44.186772 2661 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.190512 2667 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:44.190951 2667 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:44.191883 2667 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.197727 2642 raft_consensus.cc:348] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.200580 2642 raft_consensus.cc:374] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.200953 2642 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.196885 2436 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:44.202018 2642 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.204176 2643 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:44.204679 2643 ts_tablet_manager.cc:1397] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.144s user 0.008s sys 0.010s
I20260806 03:50:44.206257 2752 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:44.208177 2643 raft_consensus.cc:348] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.208945 2643 raft_consensus.cc:374] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.209210 2643 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.209854 2643 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.212276 2643 ts_tablet_manager.cc:1428] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:50:44.213820 2643 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:44.213804 2667 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.213371 2235 catalog_manager.cc:5495] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:44.216754 2667 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.218300 2644 tablet_bootstrap.cc:654] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.219911 2642 ts_tablet_manager.cc:1428] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.053s user 0.007s sys 0.000s
I20260806 03:50:44.220464 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:44.220966 2642 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:44.220492 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:44.221660 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:44.222291 2484 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.219254 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:44.223285 2584 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:44.222250 2483 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
W20260806 03:50:44.226200 2332 leader_election.cc:341] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.227468 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:44.226387 2727 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 03:50:44.228237 2584 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:44.227872 2332 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:44.230659 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:44.230816 2727 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:44.231416 2727 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:44.231855 2727 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.233672 2236 catalog_manager.cc:5495] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:44.236549 2643 tablet_bootstrap.cc:654] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.236615 2331 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:44.237960 2667 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:44.238333 2667 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:44.238688 2667 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.242583 2727 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.247125 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:44.248131 2481 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:44.248569 2331 leader_election.cc:341] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.250636 2482 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.251489 2698 consensus_queue.cc:1035] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:44.252247 2691 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.252729 2691 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.256012 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:44.256844 2377 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:44.259714 2691 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:44.259591 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:44.260851 2480 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:44.259634 2534 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:44.262440 2727 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.263118 2740 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:44.264154 2740 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:44.264536 2740 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.246887 2667 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.271817 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:44.274883 2740 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.288699 2740 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:44.290258 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:44.291158 2478 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.292484 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:44.293231 2377 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:44.288528 2332 leader_election.cc:341] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.294720 2746 consensus_queue.cc:1035] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.300835 2644 tablet_bootstrap.cc:492] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:44.301417 2644 ts_tablet_manager.cc:1397] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.224s user 0.013s sys 0.003s
I20260806 03:50:44.306902 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:44.307734 2581 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.307788 2667 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.267781 2481 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:44.311609 2332 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:44.312786 2667 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:44.314580 2667 raft_consensus.cc:686] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:44.315946 2667 consensus_queue.cc:237] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.317533 2581 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:44.318758 2331 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:44.320067 2727 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:44.320562 2727 raft_consensus.cc:686] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:44.321415 2727 consensus_queue.cc:237] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.290505 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:44.327060 2584 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.331220 2484 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:44.334213 2478 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:44.335968 2538 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:44.337105 2332 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:44.338927 2746 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:44.339592 2746 raft_consensus.cc:686] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:44.340848 2746 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.349671 2644 raft_consensus.cc:348] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.337157 2740 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:44.352803 2644 raft_consensus.cc:374] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.353478 2644 raft_consensus.cc:729] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.358541 2643 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:44.359234 2643 ts_tablet_manager.cc:1397] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.146s user 0.076s sys 0.041s
I20260806 03:50:44.354113 2740 raft_consensus.cc:686] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:44.362756 2644 consensus_queue.cc:260] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.364497 2643 raft_consensus.cc:348] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.365171 2643 raft_consensus.cc:374] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.365461 2643 raft_consensus.cc:729] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.366119 2643 consensus_queue.cc:260] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.352105 2377 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:44.360978 2740 consensus_queue.cc:237] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.370023 2644 ts_tablet_manager.cc:1428] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.068s user 0.006s sys 0.003s
I20260806 03:50:44.370878 2644 tablet_bootstrap.cc:492] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:44.372247 2643 ts_tablet_manager.cc:1428] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
W20260806 03:50:44.385851 2331 consensus_peers.cc:487] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 -> Peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Couldn't send request to peer fe3d17f4ad984b9088a268d58d87469c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:44.393064 2642 tablet_bootstrap.cc:654] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.393091 2644 tablet_bootstrap.cc:654] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.396293 2643 tablet_bootstrap.cc:492] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:44.398900 2235 catalog_manager.cc:5495] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:44.401064 2584 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:44.407469 2236 catalog_manager.cc:5495] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:44.420001 2644 tablet_bootstrap.cc:492] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:44.420660 2644 ts_tablet_manager.cc:1397] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.050s user 0.010s sys 0.003s
I20260806 03:50:44.421720 2643 tablet_bootstrap.cc:654] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.423482 2644 raft_consensus.cc:348] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.425238 2644 raft_consensus.cc:374] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.425565 2644 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.427286 2642 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:44.427790 2642 ts_tablet_manager.cc:1397] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.207s user 0.012s sys 0.002s
I20260806 03:50:44.426301 2644 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.433192 2644 ts_tablet_manager.cc:1428] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260806 03:50:44.434332 2644 tablet_bootstrap.cc:492] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:44.434490 2642 raft_consensus.cc:348] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.435300 2642 raft_consensus.cc:374] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.435567 2642 raft_consensus.cc:729] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.436836 2642 consensus_queue.cc:260] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.439059 2642 ts_tablet_manager.cc:1428] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.010s user 0.003s sys 0.002s
I20260806 03:50:44.440294 2584 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.442214 2644 tablet_bootstrap.cc:654] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.439966 2642 tablet_bootstrap.cc:492] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:44.443362 2235 catalog_manager.cc:5495] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:44.452301 2642 tablet_bootstrap.cc:654] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.461305 2643 tablet_bootstrap.cc:492] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:44.462005 2643 ts_tablet_manager.cc:1397] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.066s user 0.015s sys 0.000s
I20260806 03:50:44.464195 2740 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.465639 2643 raft_consensus.cc:348] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.466624 2643 raft_consensus.cc:374] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.466974 2643 raft_consensus.cc:729] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.467886 2643 consensus_queue.cc:260] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.471837 2643 ts_tablet_manager.cc:1428] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20260806 03:50:44.473232 2643 tablet_bootstrap.cc:492] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:44.484488 2727 consensus_queue.cc:1035] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.490378 2643 tablet_bootstrap.cc:654] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.490264 2236 catalog_manager.cc:5495] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:44.504818 2236 catalog_manager.cc:5495] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:44.496073 2727 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.510128 2727 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.529431 2746 consensus_queue.cc:1035] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.530841 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:44.533074 2727 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:44.534530 2332 leader_election.cc:341] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.535233 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:44.536901 2331 leader_election.cc:341] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.537374 2331 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:44.538616 2727 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:44.551011 2644 tablet_bootstrap.cc:492] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:44.551887 2644 ts_tablet_manager.cc:1397] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.118s user 0.030s sys 0.038s
I20260806 03:50:44.556713 2643 tablet_bootstrap.cc:492] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:44.557294 2643 ts_tablet_manager.cc:1397] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.004s
I20260806 03:50:44.557013 2644 raft_consensus.cc:348] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.557875 2644 raft_consensus.cc:374] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.558212 2644 raft_consensus.cc:729] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.559412 2644 consensus_queue.cc:260] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.561918 2643 raft_consensus.cc:348] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.562284 2644 ts_tablet_manager.cc:1428] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260806 03:50:44.562804 2643 raft_consensus.cc:374] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.563194 2643 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.563773 2644 tablet_bootstrap.cc:492] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
W20260806 03:50:44.564297 2538 consensus_peers.cc:487] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:44.563962 2643 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.567328 2643 ts_tablet_manager.cc:1428] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.010s user 0.002s sys 0.004s
I20260806 03:50:44.567664 2642 tablet_bootstrap.cc:492] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:44.568418 2642 ts_tablet_manager.cc:1397] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.129s user 0.027s sys 0.069s
I20260806 03:50:44.573186 2642 raft_consensus.cc:348] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.574090 2642 raft_consensus.cc:374] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.574445 2642 raft_consensus.cc:729] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.568595 2643 tablet_bootstrap.cc:492] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:44.578519 2642 consensus_queue.cc:260] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.580902 2642 ts_tablet_manager.cc:1428] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20260806 03:50:44.581728 2642 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:44.586388 2644 tablet_bootstrap.cc:654] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.588681 2667 consensus_queue.cc:1035] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.599745 2642 tablet_bootstrap.cc:654] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.624954 2643 tablet_bootstrap.cc:654] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.642966 2644 tablet_bootstrap.cc:492] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:44.643615 2644 ts_tablet_manager.cc:1397] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.080s user 0.021s sys 0.005s
I20260806 03:50:44.650069 2644 raft_consensus.cc:348] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.651434 2644 raft_consensus.cc:374] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.651798 2644 raft_consensus.cc:729] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.652750 2644 consensus_queue.cc:260] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.655405 2480 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.655628 2644 ts_tablet_manager.cc:1428] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260806 03:50:44.657133 2644 tablet_bootstrap.cc:492] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:44.673244 2746 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.679750 2642 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:44.680423 2642 ts_tablet_manager.cc:1397] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.099s user 0.018s sys 0.030s
I20260806 03:50:44.680919 2644 tablet_bootstrap.cc:654] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.681214 2643 tablet_bootstrap.cc:492] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:44.681768 2643 ts_tablet_manager.cc:1397] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.113s user 0.021s sys 0.014s
I20260806 03:50:44.684571 2642 raft_consensus.cc:348] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.685360 2642 raft_consensus.cc:374] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.685266 2752 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.685734 2642 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.686367 2643 raft_consensus.cc:348] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.686522 2642 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.687193 2643 raft_consensus.cc:374] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.687667 2643 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.688495 2643 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.691865 2643 ts_tablet_manager.cc:1428] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260806 03:50:44.691955 2642 ts_tablet_manager.cc:1428] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.011s user 0.003s sys 0.005s
I20260806 03:50:44.693034 2643 tablet_bootstrap.cc:492] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:44.693241 2642 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:44.706858 2643 tablet_bootstrap.cc:654] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.707067 2642 tablet_bootstrap.cc:654] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.720805 2727 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.724510 2661 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.755438 2667 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.756074 2667 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.759305 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:44.761883 2332 leader_election.cc:341] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.765223 2644 tablet_bootstrap.cc:492] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:44.766021 2644 ts_tablet_manager.cc:1397] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.109s user 0.015s sys 0.010s
I20260806 03:50:44.781481 2727 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.783771 2661 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.784513 2661 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.791561 2788 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:44.807518 2581 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:44.856861 2332 consensus_peers.cc:487] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:44.861800 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:44.863619 2331 leader_election.cc:341] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.864156 2331 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:44.865331 2690 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:44.867203 2644 raft_consensus.cc:348] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.875090 2644 raft_consensus.cc:374] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.875638 2644 raft_consensus.cc:729] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.799680 2727 consensus_queue.cc:1035] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.077s
I20260806 03:50:44.876909 2644 consensus_queue.cc:260] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.870105 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:44.878167 2788 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.870951 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:44.879469 2788 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.880596 2644 ts_tablet_manager.cc:1428] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.114s user 0.011s sys 0.015s
I20260806 03:50:44.881637 2644 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:44.882175 2752 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.882668 2752 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.884199 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:44.885432 2752 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:44.886209 2436 leader_election.cc:341] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.886375 2788 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:44.872720 2667 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.868340 2642 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:44.887466 2642 ts_tablet_manager.cc:1397] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.194s user 0.021s sys 0.025s
I20260806 03:50:44.887877 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:44.888666 2583 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:44.888815 2644 tablet_bootstrap.cc:654] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.889969 2436 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:44.890442 2642 raft_consensus.cc:348] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:44.891044 2429 leader_election.cc:341] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.869789 2661 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.891187 2642 raft_consensus.cc:374] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.891577 2429 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:44.891793 2642 raft_consensus.cc:729] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.892656 2642 consensus_queue.cc:260] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.894924 2642 ts_tablet_manager.cc:1428] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260806 03:50:44.895948 2642 tablet_bootstrap.cc:492] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:44.896905 2644 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:44.896664 2690 consensus_queue.cc:1035] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:44.899484 2644 ts_tablet_manager.cc:1397] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260806 03:50:44.902559 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:44.903349 2379 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:44.906606 2644 raft_consensus.cc:348] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.907387 2644 raft_consensus.cc:374] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:44.907684 2644 raft_consensus.cc:729] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:44.908380 2644 consensus_queue.cc:260] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.910003 2746 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.912087 2746 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:44.913113 2690 consensus_queue.cc:1035] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:44.913734 2644 ts_tablet_manager.cc:1428] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
W20260806 03:50:44.914083 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:44.914328 2746 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:44.914779 2644 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:44.911335 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:44.911561 2642 tablet_bootstrap.cc:654] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.920250 2752 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:44.920334 2661 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:44.926865 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:44.943028 2752 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:44.951738 2534 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:44.966457 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:44.967332 2740 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:44.954720 2661 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:44.968811 2644 tablet_bootstrap.cc:654] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:44.968541 2661 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:44.975623 2331 leader_election.cc:341] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:44.970175 2752 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:44.977850 2478 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:45.124142 2661 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.140493 2578 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.144765 2788 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.145418 2788 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.160756 2642 tablet_bootstrap.cc:492] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:45.161525 2642 ts_tablet_manager.cc:1397] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.266s user 0.014s sys 0.010s
I20260806 03:50:45.152633 2667 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.153987 2698 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.164546 2698 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.171876 2698 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.175419 2727 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:45.166019 2667 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.174187 2740 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260806 03:50:45.178449 2788 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:45.187566 2332 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:45.194818 2667 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.191946 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:45.210166 2644 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:45.211745 2644 ts_tablet_manager.cc:1397] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.297s user 0.020s sys 0.033s
I20260806 03:50:45.213589 2667 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260806 03:50:45.214344 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 74187.8us
I20260806 03:50:45.226845 2642 raft_consensus.cc:348] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.227895 2642 raft_consensus.cc:374] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.228273 2642 raft_consensus.cc:729] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.225911 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:45.223611 2667 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:45.231659 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:45.235996 2667 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.233184 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:45.233889 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:45.238600 2436 leader_election.cc:341] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.239006 2484 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:45.233983 2577 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:45.234262 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:45.244608 2667 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.246165 2377 raft_consensus.cc:2379] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:50:45.249608 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:45.250242 2667 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.250043 2429 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:45.251086 2661 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.230633 2644 raft_consensus.cc:348] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.252563 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:45.256371 2644 raft_consensus.cc:374] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.258482 2644 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.259248 2644 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.261603 2644 ts_tablet_manager.cc:1428] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.049s user 0.006s sys 0.000s
I20260806 03:50:45.262612 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:45.262856 2331 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:45.264199 2644 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:45.264139 2746 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:45.264726 2746 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:45.265163 2746 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.235149 2752 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.229895 2642 consensus_queue.cc:260] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:45.244643 2808 log.cc:924] Time spent T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260806 03:50:45.278978 2538 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:45.290419 2642 ts_tablet_manager.cc:1428] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.128s user 0.003s sys 0.007s
I20260806 03:50:45.292348 2642 tablet_bootstrap.cc:492] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
W20260806 03:50:45.280629 2804 log.cc:924] Time spent T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260806 03:50:45.281126 2332 leader_election.cc:341] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.288185 2752 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:45.323885 2643 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.066s sys 0.105s
W20260806 03:50:45.326396 2643 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.608s user 0.066s sys 0.105s
W20260806 03:50:45.317174 2331 leader_election.cc:341] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.334278 2644 tablet_bootstrap.cc:654] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.341401 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:45.341776 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:45.342209 2584 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.343914 2661 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:45.346529 2643 tablet_bootstrap.cc:492] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:45.347075 2643 ts_tablet_manager.cc:1397] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.654s user 0.077s sys 0.106s
I20260806 03:50:45.346681 2746 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.352536 2481 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:45.354923 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:45.357857 2379 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.364705 2740 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:45.380868 2643 raft_consensus.cc:348] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.382103 2643 raft_consensus.cc:374] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.382469 2643 raft_consensus.cc:729] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.423125 2379 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:45.428035 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:45.429056 2480 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.434605 2643 consensus_queue.cc:260] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.440438 2643 ts_tablet_manager.cc:1428] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.093s user 0.012s sys 0.002s
I20260806 03:50:45.436448 2429 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:45.434263 2584 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:45.443529 2661 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:45.445135 2661 raft_consensus.cc:686] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:45.446112 2661 consensus_queue.cc:237] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.442005 2818 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.439559 2746 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.447592 2818 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.451380 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:45.452347 2379 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:45.455937 2818 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:45.456230 2534 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:45.458580 2643 tablet_bootstrap.cc:492] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:45.460884 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:45.462563 2538 leader_election.cc:341] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.462760 2642 tablet_bootstrap.cc:654] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.466113 2818 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:45.466609 2818 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:45.468504 2818 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.466754 2480 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:45.474733 2643 tablet_bootstrap.cc:654] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.476071 2332 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:45.477028 2727 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:45.477546 2727 raft_consensus.cc:686] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:45.477452 2331 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:50:45.478634 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 174594us
I20260806 03:50:45.479321 2746 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:45.478510 2727 consensus_queue.cc:237] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.484319 2644 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:45.484417 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:45.484910 2644 ts_tablet_manager.cc:1397] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.221s user 0.015s sys 0.011s
I20260806 03:50:45.485193 2578 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.486511 2332 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:45.491963 2667 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:45.492697 2667 raft_consensus.cc:686] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:45.493891 2667 consensus_queue.cc:237] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.512941 2818 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.515144 2818 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:45.516578 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:45.517524 2379 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.526337 2644 raft_consensus.cc:348] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.527494 2644 raft_consensus.cc:374] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.536450 2644 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.539709 2644 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.535151 2578 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:45.531541 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:45.544153 2644 ts_tablet_manager.cc:1428] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.059s user 0.009s sys 0.000s
W20260806 03:50:45.545287 2538 leader_election.cc:341] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:45.548631 2331 leader_election.cc:341] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.553362 2642 tablet_bootstrap.cc:492] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:45.553925 2642 ts_tablet_manager.cc:1397] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.262s user 0.015s sys 0.033s
I20260806 03:50:45.554904 2379 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:45.555389 2233 catalog_manager.cc:5495] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:45.557770 2534 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:45.558774 2818 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:45.561642 2818 raft_consensus.cc:686] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:45.562570 2818 consensus_queue.cc:237] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.560348 2642 raft_consensus.cc:348] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.563901 2642 raft_consensus.cc:374] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.564306 2642 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.569558 2642 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.573590 2644 tablet_bootstrap.cc:492] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:45.586500 2642 ts_tablet_manager.cc:1428] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.032s user 0.009s sys 0.000s
I20260806 03:50:45.588315 2642 tablet_bootstrap.cc:492] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
W20260806 03:50:45.602041 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 43997us
I20260806 03:50:45.630084 2642 tablet_bootstrap.cc:654] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.665161 2698 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.665972 2698 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.676448 2698 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.676899 2644 tablet_bootstrap.cc:654] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.678026 2235 catalog_manager.cc:5495] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:45.681663 2643 tablet_bootstrap.cc:492] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:45.678026 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:45.682543 2643 ts_tablet_manager.cc:1397] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.224s user 0.018s sys 0.007s
I20260806 03:50:45.683212 2661 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.683830 2661 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:45.685179 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 59720.3us
I20260806 03:50:45.685791 2643 raft_consensus.cc:348] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.686589 2643 raft_consensus.cc:374] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.686875 2643 raft_consensus.cc:729] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
W20260806 03:50:45.687163 2332 leader_election.cc:341] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.687784 2643 consensus_queue.cc:260] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.690039 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:45.690531 2643 ts_tablet_manager.cc:1428] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 03:50:45.690912 2379 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:45.691617 2643 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:45.693120 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:45.695060 2436 leader_election.cc:341] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.695765 2661 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.697001 2429 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:45.700636 2643 tablet_bootstrap.cc:654] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.699245 2661 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:45.702251 2661 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:45.702718 2661 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.713826 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:45.719144 2644 tablet_bootstrap.cc:492] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:45.719739 2644 ts_tablet_manager.cc:1397] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.146s user 0.019s sys 0.018s
W20260806 03:50:45.721549 2331 leader_election.cc:341] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.721920 2642 tablet_bootstrap.cc:492] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:45.721437 2661 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.722429 2642 ts_tablet_manager.cc:1397] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.134s user 0.014s sys 0.013s
I20260806 03:50:45.723174 2644 raft_consensus.cc:348] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.724045 2644 raft_consensus.cc:374] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.724327 2644 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.722134 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:45.725524 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:45.725937 2835 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:45.726193 2379 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.725297 2642 raft_consensus.cc:348] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.727345 2642 raft_consensus.cc:374] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.727707 2643 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:45.727675 2642 raft_consensus.cc:729] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.728368 2643 ts_tablet_manager.cc:1397] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.001s
I20260806 03:50:45.728641 2642 consensus_queue.cc:260] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.731215 2642 ts_tablet_manager.cc:1428] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260806 03:50:45.732301 2642 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:45.732110 2643 raft_consensus.cc:348] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.732869 2643 raft_consensus.cc:374] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.733163 2643 raft_consensus.cc:729] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.725010 2644 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.736716 2643 consensus_queue.cc:260] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.737426 2379 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:45.737655 2752 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.738334 2644 ts_tablet_manager.cc:1428] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.018s user 0.006s sys 0.001s
I20260806 03:50:45.738198 2752 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.739428 2644 tablet_bootstrap.cc:492] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:45.739753 2642 tablet_bootstrap.cc:654] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.743326 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:45.746248 2644 tablet_bootstrap.cc:654] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.750612 2661 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.750777 2429 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:45.751438 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:45.753361 2643 ts_tablet_manager.cc:1428] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260806 03:50:45.753438 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:45.754554 2643 tablet_bootstrap.cc:492] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:45.755390 2436 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:50:45.756429 2436 leader_election.cc:341] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.760254 2811 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:45.760247 2661 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:45.760942 2811 raft_consensus.cc:686] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:45.748365 2752 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.763432 2835 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.761686 2811 consensus_queue.cc:237] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.763974 2835 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.776301 2835 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.778049 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:45.780402 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:45.784540 2332 leader_election.cc:341] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:45.785575 2331 leader_election.cc:341] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.786013 2331 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:45.787460 2835 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:45.788584 2642 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:45.789318 2642 ts_tablet_manager.cc:1397] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.031s
I20260806 03:50:45.801501 2642 raft_consensus.cc:348] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.804008 2642 raft_consensus.cc:374] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.804430 2642 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.805202 2642 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.807909 2642 ts_tablet_manager.cc:1428] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.018s user 0.008s sys 0.000s
I20260806 03:50:45.808983 2642 tablet_bootstrap.cc:492] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:45.820287 2643 tablet_bootstrap.cc:654] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.821621 2644 tablet_bootstrap.cc:492] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:45.822141 2644 ts_tablet_manager.cc:1397] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.083s user 0.020s sys 0.019s
I20260806 03:50:45.824930 2644 raft_consensus.cc:348] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.818423 2232 catalog_manager.cc:5495] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:45.826880 2644 raft_consensus.cc:374] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.827211 2644 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.829243 2644 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.830551 2642 tablet_bootstrap.cc:654] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.834900 2644 ts_tablet_manager.cc:1428] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260806 03:50:45.836191 2644 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:45.836575 2643 tablet_bootstrap.cc:492] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:45.836978 2818 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.837543 2818 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.838264 2811 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.838783 2811 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.839854 2818 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:45.841135 2811 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.841449 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:45.841782 2643 ts_tablet_manager.cc:1397] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.087s user 0.011s sys 0.006s
I20260806 03:50:45.843377 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:45.843824 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:45.845041 2643 raft_consensus.cc:348] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:45.845808 2538 leader_election.cc:341] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.845991 2643 raft_consensus.cc:374] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.847242 2643 raft_consensus.cc:729] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.847383 2436 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:45.847393 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:45.847998 2643 consensus_queue.cc:260] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.849018 2379 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:45.850507 2643 ts_tablet_manager.cc:1428] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.008s user 0.003s sys 0.002s
I20260806 03:50:45.850986 2534 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:45.851558 2643 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:45.852509 2818 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:45.852934 2818 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:45.853283 2818 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.855288 2644 tablet_bootstrap.cc:654] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.859606 2643 tablet_bootstrap.cc:654] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.861723 2752 consensus_queue.cc:1035] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:45.862319 2642 tablet_bootstrap.cc:492] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:45.859700 2811 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:45.862824 2642 ts_tablet_manager.cc:1397] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.002s
I20260806 03:50:45.865792 2642 raft_consensus.cc:348] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.866583 2642 raft_consensus.cc:374] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.866997 2642 raft_consensus.cc:729] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.867798 2642 consensus_queue.cc:260] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.870487 2642 ts_tablet_manager.cc:1428] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 03:50:45.872488 2642 tablet_bootstrap.cc:492] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:45.887699 2642 tablet_bootstrap.cc:654] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.896756 2235 catalog_manager.cc:5495] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:45.902195 2642 tablet_bootstrap.cc:492] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:45.902983 2642 ts_tablet_manager.cc:1397] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.005s
I20260806 03:50:45.903609 2698 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:45.904151 2698 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.906816 2698 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:45.908864 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:45.912734 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:45.913373 2332 leader_election.cc:341] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.912726 2581 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.914513 2818 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.918794 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:45.919775 2379 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.907363 2642 raft_consensus.cc:348] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.920441 2642 raft_consensus.cc:374] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.920763 2642 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
W20260806 03:50:45.921753 2331 leader_election.cc:341] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.921456 2642 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.922240 2331 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:45.923024 2667 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:45.923951 2642 ts_tablet_manager.cc:1428] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.021s user 0.008s sys 0.000s
I20260806 03:50:45.925006 2642 tablet_bootstrap.cc:492] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:45.921519 2811 consensus_queue.cc:1035] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:45.928864 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:45.929520 2818 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:50:45.930228 2538 leader_election.cc:341] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:45.931313 2642 tablet_bootstrap.cc:654] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:45.932626 2667 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:45.934190 2379 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:45.935598 2534 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:45.938069 2817 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:45.938598 2817 raft_consensus.cc:686] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:45.939610 2817 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.946494 2236 catalog_manager.cc:5495] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:45.948680 2481 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:45.962225 2835 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:45.963057 2643 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:45.963620 2643 ts_tablet_manager.cc:1397] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.112s user 0.022s sys 0.010s
I20260806 03:50:45.975245 2835 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:45.986951 2643 raft_consensus.cc:348] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.987888 2643 raft_consensus.cc:374] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.988209 2643 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.994045 2642 tablet_bootstrap.cc:492] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:45.994632 2642 ts_tablet_manager.cc:1397] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.010s
I20260806 03:50:45.996729 2835 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:45.997438 2642 raft_consensus.cc:348] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:45.998198 2642 raft_consensus.cc:374] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:45.998498 2642 raft_consensus.cc:729] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:45.999302 2642 consensus_queue.cc:260] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.000209 2644 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:46.000846 2644 ts_tablet_manager.cc:1397] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.165s user 0.017s sys 0.019s
I20260806 03:50:46.001776 2642 ts_tablet_manager.cc:1428] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260806 03:50:46.003077 2642 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.004918 2644 raft_consensus.cc:348] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.005841 2644 raft_consensus.cc:374] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.006161 2644 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.021315 2643 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.022199 2642 tablet_bootstrap.cc:654] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.025130 2643 ts_tablet_manager.cc:1428] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.061s user 0.009s sys 0.000s
I20260806 03:50:46.027297 2643 tablet_bootstrap.cc:492] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:46.027323 2644 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.032920 2835 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.034006 2835 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.035142 2644 ts_tablet_manager.cc:1428] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.034s user 0.007s sys 0.001s
I20260806 03:50:46.036228 2835 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.037966 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:46.038652 2584 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:46.038707 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.040992 2811 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.041663 2811 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:46.044668 2332 leader_election.cc:341] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.045105 2811 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.046062 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.046944 2644 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:46.046929 2379 raft_consensus.cc:2379] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:46.051970 2331 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:46.052336 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:46.059906 2698 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:46.105644 2817 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.106377 2817 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.082885 2698 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:46.131290 2817 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.130460 2436 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:46.133625 2818 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.134711 2811 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.134440 2818 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.147537 2643 tablet_bootstrap.cc:654] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.211063 2642 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:46.214421 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.215091 2642 ts_tablet_manager.cc:1397] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.212s user 0.025s sys 0.044s
I20260806 03:50:46.151546 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.218075 2642 raft_consensus.cc:348] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.218346 2481 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:46.218745 2642 raft_consensus.cc:374] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.219025 2642 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.219704 2642 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.221290 2538 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:46.222098 2642 ts_tablet_manager.cc:1428] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260806 03:50:46.222226 2817 raft_consensus.cc:2781] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:46.222647 2817 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:46.222990 2817 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.223107 2642 tablet_bootstrap.cc:492] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.129959 2698 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.162994 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.231954 2379 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
W20260806 03:50:46.232349 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 104683us
I20260806 03:50:46.172065 2818 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.237816 2480 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.238750 2642 tablet_bootstrap.cc:654] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.242522 2691 consensus_queue.cc:1035] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 03:50:46.244374 2534 leader_election.cc:341] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.231594 2817 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.252252 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:46.250437 2538 consensus_peers.cc:487] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:46.248632 2811 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:46.252530 2817 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:50:46.260396 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260806 03:50:46.260752 2538 leader_election.cc:341] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.261211 2538 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:46.264675 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:46.264961 2644 tablet_bootstrap.cc:654] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.265384 2377 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.265235 2818 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:46.267303 2691 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.272512 2691 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.274799 2691 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.276494 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.278199 2375 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:46.269093 2817 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.280622 2534 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:46.288452 2642 tablet_bootstrap.cc:492] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:46.288977 2642 ts_tablet_manager.cc:1397] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.066s user 0.019s sys 0.001s
I20260806 03:50:46.290964 2644 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:46.291666 2644 ts_tablet_manager.cc:1397] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.245s user 0.012s sys 0.006s
I20260806 03:50:46.290194 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:46.294502 2481 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.296905 2642 raft_consensus.cc:348] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.297721 2642 raft_consensus.cc:374] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.297982 2642 raft_consensus.cc:729] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.298113 2886 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:46.290553 2817 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.298695 2886 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:46.292373 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.299353 2886 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.298744 2642 consensus_queue.cc:260] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:46.303568 2538 leader_election.cc:341] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.313992 2578 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.332626 2642 ts_tablet_manager.cc:1428] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.043s user 0.006s sys 0.003s
I20260806 03:50:46.341627 2643 tablet_bootstrap.cc:492] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:46.342288 2643 ts_tablet_manager.cc:1397] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.315s user 0.026s sys 0.006s
I20260806 03:50:46.342182 2817 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.346233 2642 tablet_bootstrap.cc:492] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.347797 2644 raft_consensus.cc:348] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.348492 2644 raft_consensus.cc:374] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.348752 2644 raft_consensus.cc:729] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.301433 2236 catalog_manager.cc:5495] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:46.344682 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.350867 2644 consensus_queue.cc:260] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.353909 2644 ts_tablet_manager.cc:1428] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.062s user 0.007s sys 0.005s
I20260806 03:50:46.345034 2698 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.355016 2644 tablet_bootstrap.cc:492] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:46.355123 2643 raft_consensus.cc:348] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.361783 2643 raft_consensus.cc:374] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.362162 2643 raft_consensus.cc:729] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.351949 2379 raft_consensus.cc:2379] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 1: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:50:46.363133 2643 consensus_queue.cc:260] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.378917 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:46.382728 2643 ts_tablet_manager.cc:1428] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.040s user 0.005s sys 0.002s
I20260806 03:50:46.383599 2643 tablet_bootstrap.cc:492] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:46.386021 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.388494 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:46.389240 2584 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.391149 2752 consensus_queue.cc:1035] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:46.393658 2818 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:46.394815 2538 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:46.395843 2377 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
W20260806 03:50:46.393870 2332 leader_election.cc:341] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.400126 2481 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:46.380524 2886 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.405675 2886 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.397074 2698 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.413158 2538 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:46.412981 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:46.422341 2644 tablet_bootstrap.cc:654] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.424137 2376 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.426167 2817 raft_consensus.cc:2781] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:46.426688 2817 raft_consensus.cc:686] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:46.427402 2818 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:46.470810 2811 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.437011 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:46.478632 2811 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.483605 2835 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.434968 2817 consensus_queue.cc:237] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.484220 2835 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.483767 2811 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.486406 2835 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.488147 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.453801 2698 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.501572 2642 tablet_bootstrap.cc:654] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.439134 2584 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
W20260806 03:50:46.503599 2538 leader_election.cc:341] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.439486 2818 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.057s
I20260806 03:50:46.515172 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:46.517225 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:46.519359 2332 leader_election.cc:341] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.523841 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:46.524864 2429 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:46.512542 2698 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.526669 2811 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:46.528299 2643 tablet_bootstrap.cc:654] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.435452 2752 consensus_queue.cc:1035] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.064s
I20260806 03:50:46.528949 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.530236 2698 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.531566 2331 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
W20260806 03:50:46.533206 2331 leader_election.cc:341] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.533452 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:46.533669 2331 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:46.534456 2746 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:46.534951 2746 raft_consensus.cc:686] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:46.535497 2836 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:46.535830 2746 consensus_queue.cc:237] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:46.540917 2332 leader_election.cc:341] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.523972 2376 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
W20260806 03:50:46.550395 2331 leader_election.cc:341] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.559430 2534 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:46.574935 2892 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:46.575752 2892 raft_consensus.cc:686] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:46.586544 2644 tablet_bootstrap.cc:492] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:46.587278 2644 ts_tablet_manager.cc:1397] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.232s user 0.012s sys 0.014s
I20260806 03:50:46.558607 2331 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:46.585136 2236 catalog_manager.cc:5495] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:46.594640 2746 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:46.590732 2892 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.600952 2235 catalog_manager.cc:5495] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:46.610957 2642 tablet_bootstrap.cc:492] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
W20260806 03:50:46.592602 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 45622.2us
I20260806 03:50:46.611719 2642 ts_tablet_manager.cc:1397] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.271s user 0.007s sys 0.009s
I20260806 03:50:46.619822 2642 raft_consensus.cc:348] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.620898 2642 raft_consensus.cc:374] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.621146 2642 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.623361 2642 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.627452 2644 raft_consensus.cc:348] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.628327 2644 raft_consensus.cc:374] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.628597 2644 raft_consensus.cc:729] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.629279 2644 consensus_queue.cc:260] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.632579 2644 ts_tablet_manager.cc:1428] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.045s user 0.006s sys 0.001s
I20260806 03:50:46.633685 2644 tablet_bootstrap.cc:492] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:46.634227 2643 tablet_bootstrap.cc:492] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:46.634730 2643 ts_tablet_manager.cc:1397] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.251s user 0.010s sys 0.008s
I20260806 03:50:46.638419 2643 raft_consensus.cc:348] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.638967 2643 raft_consensus.cc:374] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.639271 2643 raft_consensus.cc:729] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.640398 2643 consensus_queue.cc:260] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.641551 2642 ts_tablet_manager.cc:1428] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260806 03:50:46.642553 2642 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.647440 2643 ts_tablet_manager.cc:1428] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.012s user 0.007s sys 0.002s
I20260806 03:50:46.648669 2643 tablet_bootstrap.cc:492] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:46.658197 2644 tablet_bootstrap.cc:654] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.667507 2642 tablet_bootstrap.cc:654] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.670291 2643 tablet_bootstrap.cc:654] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.687628 2642 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:46.689175 2642 ts_tablet_manager.cc:1397] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.000s
I20260806 03:50:46.692582 2642 raft_consensus.cc:348] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.696655 2642 raft_consensus.cc:374] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.697492 2642 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.698589 2642 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.703066 2642 ts_tablet_manager.cc:1428] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260806 03:50:46.706090 2642 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.714607 2643 tablet_bootstrap.cc:492] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:46.714846 2644 tablet_bootstrap.cc:492] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:46.715204 2643 ts_tablet_manager.cc:1397] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.020s
I20260806 03:50:46.715354 2644 ts_tablet_manager.cc:1397] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.082s user 0.020s sys 0.018s
I20260806 03:50:46.715857 2892 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.716830 2642 tablet_bootstrap.cc:654] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.716552 2892 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.718281 2643 raft_consensus.cc:348] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.719004 2643 raft_consensus.cc:374] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.718761 2644 raft_consensus.cc:348] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.719303 2643 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.719543 2644 raft_consensus.cc:374] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.719851 2644 raft_consensus.cc:729] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.719880 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.720142 2643 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.720714 2644 consensus_queue.cc:260] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:46.721599 2534 leader_election.cc:341] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.723706 2644 ts_tablet_manager.cc:1428] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260806 03:50:46.723716 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.725164 2644 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:46.726684 2643 ts_tablet_manager.cc:1428] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.011s user 0.005s sys 0.004s
I20260806 03:50:46.727792 2643 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
W20260806 03:50:46.728355 2538 leader_election.cc:341] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.728837 2538 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:46.730976 2817 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:46.734476 2644 tablet_bootstrap.cc:654] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.734684 2643 tablet_bootstrap.cc:654] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.736435 2892 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.740123 2642 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:46.740911 2642 ts_tablet_manager.cc:1397] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.006s
I20260806 03:50:46.744719 2642 raft_consensus.cc:348] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.746007 2642 raft_consensus.cc:374] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.746433 2642 raft_consensus.cc:729] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.747412 2642 consensus_queue.cc:260] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.751976 2642 ts_tablet_manager.cc:1428] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260806 03:50:46.753057 2642 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.755172 2644 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:46.754299 2235 catalog_manager.cc:5495] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:46.755784 2644 ts_tablet_manager.cc:1397] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.031s user 0.013s sys 0.012s
I20260806 03:50:46.757886 2643 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:46.758356 2643 ts_tablet_manager.cc:1397] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.008s
I20260806 03:50:46.758716 2644 raft_consensus.cc:348] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.759465 2644 raft_consensus.cc:374] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.759720 2644 raft_consensus.cc:729] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.760320 2643 raft_consensus.cc:348] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.760890 2643 raft_consensus.cc:374] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.760514 2644 consensus_queue.cc:260] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.761159 2643 raft_consensus.cc:729] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.761839 2643 consensus_queue.cc:260] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.764313 2642 tablet_bootstrap.cc:654] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.764572 2644 ts_tablet_manager.cc:1428] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:50:46.765679 2644 tablet_bootstrap.cc:492] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:46.769891 2643 ts_tablet_manager.cc:1428] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.011s user 0.007s sys 0.001s
I20260806 03:50:46.770853 2643 tablet_bootstrap.cc:492] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:46.772185 2644 tablet_bootstrap.cc:654] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.776125 2642 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:46.776641 2642 ts_tablet_manager.cc:1397] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.010s
I20260806 03:50:46.777890 2643 tablet_bootstrap.cc:654] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.779287 2642 raft_consensus.cc:348] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.780064 2642 raft_consensus.cc:374] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.780344 2642 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.781049 2642 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.783694 2642 ts_tablet_manager.cc:1428] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260806 03:50:46.784716 2642 tablet_bootstrap.cc:492] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.790730 2642 tablet_bootstrap.cc:654] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.795249 2484 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.800014 2642 tablet_bootstrap.cc:492] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:46.800549 2642 ts_tablet_manager.cc:1397] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260806 03:50:46.803517 2642 raft_consensus.cc:348] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.804267 2642 raft_consensus.cc:374] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.804538 2642 raft_consensus.cc:729] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.805190 2642 consensus_queue.cc:260] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.808342 2642 ts_tablet_manager.cc:1428] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:50:46.809324 2642 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.810204 2643 tablet_bootstrap.cc:492] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:46.811806 2643 ts_tablet_manager.cc:1397] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.009s
I20260806 03:50:46.815372 2642 tablet_bootstrap.cc:654] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.821146 2811 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.821380 2836 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.822103 2811 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.822173 2836 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.824791 2892 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.825336 2892 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.826489 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.827255 2481 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:46.828975 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.829897 2375 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:46.829991 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:46.830745 2892 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.831030 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.831733 2376 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:46.832221 2429 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:46.833171 2534 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:46.833583 2897 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:46.834033 2892 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:46.834874 2892 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:46.835557 2892 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.836220 2836 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.830749 2578 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:46.838584 2331 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:46.834081 2897 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:46.840509 2897 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.840695 2667 raft_consensus.cc:2781] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:46.840600 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.841253 2667 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:46.842420 2667 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.864001 2643 raft_consensus.cc:348] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.865162 2643 raft_consensus.cc:374] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.865504 2643 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.872110 2643 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.886103 2642 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:46.886747 2642 ts_tablet_manager.cc:1397] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.078s user 0.009s sys 0.011s
I20260806 03:50:46.893035 2643 ts_tablet_manager.cc:1428] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.079s user 0.009s sys 0.000s
I20260806 03:50:46.891247 2642 raft_consensus.cc:348] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.893946 2642 raft_consensus.cc:374] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:46.894321 2642 raft_consensus.cc:729] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:46.894286 2643 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:46.902825 2698 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.903402 2698 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.895609 2642 consensus_queue.cc:260] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.908162 2698 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.909004 2836 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.909718 2836 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.910398 2892 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.914228 2811 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.915524 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:46.915860 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:46.916421 2578 raft_consensus.cc:2379] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:46.923380 2897 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.934890 2836 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.936347 2481 raft_consensus.cc:2379] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 1: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:50:46.948275 2897 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:46.962764 2817 consensus_queue.cc:1035] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:46.968966 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:46.970525 2376 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:46.973266 2642 ts_tablet_manager.cc:1428] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.086s user 0.009s sys 0.000s
I20260806 03:50:46.942123 2643 tablet_bootstrap.cc:654] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:46.940093 2667 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.943233 2691 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:46.978592 2691 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:46.981535 2691 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.944111 2892 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:46.939538 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:46.983698 2578 raft_consensus.cc:2379] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:46.984372 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:46.987051 2667 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:46.988729 2538 leader_election.cc:341] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:46.993242 2642 tablet_bootstrap.cc:492] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:46.998499 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:46.999526 2375 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:47.001267 2886 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.001754 2892 consensus_queue.cc:1035] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.001883 2886 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.004769 2886 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:47.007596 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:47.009706 2538 leader_election.cc:341] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.010010 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:47.010996 2578 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:47.013180 2534 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:47.014995 2892 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:47.015623 2892 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:47.023087 2892 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.023386 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:47.026499 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:47.021582 2376 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
W20260806 03:50:47.032941 2534 leader_election.cc:341] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.033473 2534 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:47.039454 2534 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:47.040648 2691 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:47.041177 2691 raft_consensus.cc:686] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:47.041217 2886 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:47.042146 2644 tablet_bootstrap.cc:492] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:47.042675 2644 ts_tablet_manager.cc:1397] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.277s user 0.075s sys 0.136s
I20260806 03:50:47.042069 2691 consensus_queue.cc:237] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.045279 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.050647 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.052310 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:47.053014 2484 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.048085 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:47.049710 2331 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:47.053814 2578 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.052424 2644 raft_consensus.cc:348] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.056236 2644 raft_consensus.cc:374] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.056598 2644 raft_consensus.cc:729] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.057008 2642 tablet_bootstrap.cc:654] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.057565 2644 consensus_queue.cc:260] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.073139 2644 ts_tablet_manager.cc:1428] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.030s user 0.006s sys 0.001s
I20260806 03:50:47.058768 2892 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.061591 2667 raft_consensus.cc:2781] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:47.074956 2667 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:47.075325 2667 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.080060 2643 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:47.066635 2817 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 03:50:47.084241 2332 leader_election.cc:341] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.084901 2481 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.080924 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:47.085755 2379 raft_consensus.cc:2379] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:50:47.066220 2331 leader_election.cc:341] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.083737 2667 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.087069 2643 ts_tablet_manager.cc:1397] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.193s user 0.016s sys 0.003s
I20260806 03:50:47.074534 2644 tablet_bootstrap.cc:492] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:47.088908 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:47.089655 2379 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.092252 2892 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:47.089018 2578 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:47.093950 2899 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.093502 2667 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:47.094518 2899 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:47.096892 2332 leader_election.cc:341] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.097849 2332 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:47.103122 2643 raft_consensus.cc:348] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.105274 2643 raft_consensus.cc:374] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.105661 2643 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.108511 2643 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.103984 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:47.095997 2484 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:47.113905 2429 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:47.116305 2583 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.116627 2332 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:47.118794 2920 raft_consensus.cc:2781] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:47.119402 2920 raft_consensus.cc:686] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:47.120889 2899 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:47.120572 2920 consensus_queue.cc:237] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.110455 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:47.122355 2482 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.127189 2897 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:50:47.128472 2892 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.129020 2892 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.129455 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:47.131228 2892 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:47.132131 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:47.132934 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:47.132128 2332 leader_election.cc:341] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.133618 2579 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
W20260806 03:50:47.135064 2538 consensus_peers.cc:487] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:47.135584 2921 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:50:47.135092 2534 leader_election.cc:341] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.136210 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:47.136938 2376 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:47.137266 2436 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:47.139065 2899 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:47.139545 2899 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:47.139984 2899 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.143985 2643 ts_tablet_manager.cc:1428] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.057s user 0.007s sys 0.001s
I20260806 03:50:47.145166 2643 tablet_bootstrap.cc:492] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:47.149358 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.158473 2922 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.163259 2379 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:47.164881 2534 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:47.165959 2583 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:47.166632 2892 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:47.167343 2482 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:47.169148 2892 raft_consensus.cc:686] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:50:47.169837 2538 leader_election.cc:341] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.170223 2331 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:47.170356 2538 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:47.170636 2836 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:47.174922 2921 raft_consensus.cc:2781] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:47.175501 2921 raft_consensus.cc:686] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:47.180241 2921 consensus_queue.cc:237] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.174592 2691 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:47.170183 2892 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.196790 2644 tablet_bootstrap.cc:654] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.197736 2642 tablet_bootstrap.cc:492] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:47.198315 2642 ts_tablet_manager.cc:1397] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.205s user 0.025s sys 0.021s
I20260806 03:50:47.196593 2232 catalog_manager.cc:5495] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:47.238541 2236 catalog_manager.cc:5495] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:47.271641 2642 raft_consensus.cc:348] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.272552 2642 raft_consensus.cc:374] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.272853 2642 raft_consensus.cc:729] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.273510 2642 consensus_queue.cc:260] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.275362 2642 ts_tablet_manager.cc:1428] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.077s user 0.006s sys 0.001s
I20260806 03:50:47.276443 2642 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:47.279803 2643 tablet_bootstrap.cc:654] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.298751 2899 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.301275 2899 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:47.302963 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:47.305249 2644 tablet_bootstrap.cc:492] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:47.305732 2644 ts_tablet_manager.cc:1397] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.231s user 0.011s sys 0.010s
I20260806 03:50:47.303243 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:47.308844 2375 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.306607 2581 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.320683 2375 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:47.323863 2642 tablet_bootstrap.cc:654] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.322324 2429 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:47.325193 2899 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:47.325943 2899 raft_consensus.cc:686] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:47.328100 2899 consensus_queue.cc:237] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.331648 2581 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:47.337136 2892 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.337996 2892 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.341720 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.340149 2892 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:47.346982 2644 raft_consensus.cc:348] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.347795 2644 raft_consensus.cc:374] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.348093 2644 raft_consensus.cc:729] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.348693 2644 consensus_queue.cc:260] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:47.356722 2538 leader_election.cc:341] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.360685 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:47.361522 2375 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:47.364313 2644 ts_tablet_manager.cc:1428] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.058s user 0.005s sys 0.004s
I20260806 03:50:47.365548 2644 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:47.365036 2235 catalog_manager.cc:5495] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:47.375149 2534 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:47.378525 2892 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:47.379037 2892 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:47.379419 2892 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.382594 2644 tablet_bootstrap.cc:654] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.370850 2236 catalog_manager.cc:5495] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:47.391355 2643 tablet_bootstrap.cc:492] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:47.392198 2643 ts_tablet_manager.cc:1397] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.247s user 0.024s sys 0.000s
I20260806 03:50:47.395255 2643 raft_consensus.cc:348] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.396298 2643 raft_consensus.cc:374] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.396672 2643 raft_consensus.cc:729] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.400388 2644 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:47.400662 2232 catalog_manager.cc:5495] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:47.404627 2644 ts_tablet_manager.cc:1397] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.006s
I20260806 03:50:47.403010 2642 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:47.406090 2642 ts_tablet_manager.cc:1397] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.130s user 0.023s sys 0.044s
I20260806 03:50:47.407573 2644 raft_consensus.cc:348] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.402951 2892 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.408386 2644 raft_consensus.cc:374] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.408782 2644 raft_consensus.cc:729] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.409462 2644 consensus_queue.cc:260] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.410259 2892 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:47.411185 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:47.411376 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:47.411957 2379 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:47.413142 2538 leader_election.cc:341] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.415721 2644 ts_tablet_manager.cc:1428] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260806 03:50:47.411070 2642 raft_consensus.cc:348] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.416195 2642 raft_consensus.cc:374] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.416495 2642 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.412667 2643 consensus_queue.cc:260] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.416723 2644 tablet_bootstrap.cc:492] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:47.417991 2642 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.419235 2643 ts_tablet_manager.cc:1428] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.027s user 0.005s sys 0.002s
I20260806 03:50:47.420182 2642 ts_tablet_manager.cc:1428] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260806 03:50:47.420228 2643 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:47.420472 2921 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.421213 2642 tablet_bootstrap.cc:492] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:47.420974 2921 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.424522 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:47.425252 2581 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:47.425981 2921 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:47.427217 2331 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:47.427664 2643 tablet_bootstrap.cc:654] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.428555 2642 tablet_bootstrap.cc:654] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.428539 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.429315 2478 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:47.430974 2920 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.431488 2920 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.436275 2920 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:47.436913 2921 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:47.437369 2921 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:47.437806 2921 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.442931 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.443199 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:47.444710 2331 leader_election.cc:341] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:47.446506 2332 leader_election.cc:341] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.448807 2332 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:47.450042 2920 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:47.451349 2921 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.458269 2379 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:47.459790 2534 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:47.461153 2817 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:47.461963 2817 raft_consensus.cc:686] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:47.462918 2817 consensus_queue.cc:237] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.466718 2644 tablet_bootstrap.cc:654] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.470559 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:47.471231 2478 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:47.474045 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:47.482283 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:47.482910 2578 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.484359 2642 tablet_bootstrap.cc:492] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:47.484894 2642 ts_tablet_manager.cc:1397] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.064s user 0.024s sys 0.019s
I20260806 03:50:47.487676 2642 raft_consensus.cc:348] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.488441 2642 raft_consensus.cc:374] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.488703 2642 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.488873 2578 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:47.489387 2642 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.492901 2642 ts_tablet_manager.cc:1428] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260806 03:50:47.490257 2331 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:47.493994 2642 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:47.490218 2921 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:47.494944 2236 catalog_manager.cc:5495] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:47.496554 2478 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:47.494980 2921 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:47.501344 2921 raft_consensus.cc:686] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:47.502259 2921 consensus_queue.cc:237] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.506825 2642 tablet_bootstrap.cc:654] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.508000 2643 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:47.508514 2643 ts_tablet_manager.cc:1397] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.088s user 0.030s sys 0.022s
I20260806 03:50:47.511984 2643 raft_consensus.cc:348] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.512821 2643 raft_consensus.cc:374] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.513131 2643 raft_consensus.cc:729] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.513970 2817 consensus_queue.cc:1035] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.514571 2922 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.515034 2922 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.517258 2922 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:47.513919 2643 consensus_queue.cc:260] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.521037 2642 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:47.520730 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:47.538120 2642 ts_tablet_manager.cc:1397] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.009s
I20260806 03:50:47.526314 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.532294 2817 consensus_queue.cc:1035] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.542111 2643 ts_tablet_manager.cc:1428] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.033s user 0.003s sys 0.004s
I20260806 03:50:47.543416 2643 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:47.542958 2236 catalog_manager.cc:5495] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:47.542362 2642 raft_consensus.cc:348] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.545815 2642 raft_consensus.cc:374] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260806 03:50:47.546021 2331 leader_election.cc:341] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.546244 2642 raft_consensus.cc:729] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.546962 2642 consensus_queue.cc:260] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.569357 2642 ts_tablet_manager.cc:1428] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.031s user 0.008s sys 0.000s
I20260806 03:50:47.570350 2642 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
W20260806 03:50:47.570329 2332 leader_election.cc:341] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.571560 2332 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:47.573033 2922 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:47.573789 2644 tablet_bootstrap.cc:492] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:47.574288 2644 ts_tablet_manager.cc:1397] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.158s user 0.013s sys 0.014s
I20260806 03:50:47.575277 2643 tablet_bootstrap.cc:654] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.586177 2642 tablet_bootstrap.cc:654] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.589438 2644 raft_consensus.cc:348] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.590337 2644 raft_consensus.cc:374] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.590683 2644 raft_consensus.cc:729] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.596720 2667 consensus_queue.cc:1035] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:47.604058 2817 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.614413 2642 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:47.613237 2644 consensus_queue.cc:260] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.615289 2642 ts_tablet_manager.cc:1397] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.009s
I20260806 03:50:47.620872 2644 ts_tablet_manager.cc:1428] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.046s user 0.007s sys 0.001s
I20260806 03:50:47.621325 2886 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.623939 2642 raft_consensus.cc:348] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.624799 2642 raft_consensus.cc:374] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.625149 2642 raft_consensus.cc:729] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.625901 2642 consensus_queue.cc:260] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.628362 2642 ts_tablet_manager.cc:1428] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260806 03:50:47.628449 2643 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:47.629460 2643 ts_tablet_manager.cc:1397] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.086s user 0.020s sys 0.027s
I20260806 03:50:47.629495 2642 tablet_bootstrap.cc:492] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:47.626456 2644 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:47.632349 2643 raft_consensus.cc:348] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.633270 2643 raft_consensus.cc:374] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.633589 2643 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.638559 2642 tablet_bootstrap.cc:654] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.637537 2643 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.644207 2644 tablet_bootstrap.cc:654] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.644657 2643 ts_tablet_manager.cc:1428] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260806 03:50:47.648595 2643 tablet_bootstrap.cc:492] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:47.660995 2643 tablet_bootstrap.cc:654] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.676577 2921 consensus_queue.cc:1035] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 03:50:47.682718 2482 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.684211 2481 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.691213 2922 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260806 03:50:47.705581 2920 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 03:50:47.708259 2886 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.709015 2886 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.719300 2886 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:47.721359 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.726168 2642 tablet_bootstrap.cc:492] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:47.726943 2642 ts_tablet_manager.cc:1397] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.098s user 0.019s sys 0.021s
I20260806 03:50:47.731375 2644 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:47.731927 2644 ts_tablet_manager.cc:1397] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.106s user 0.019s sys 0.048s
I20260806 03:50:47.734529 2644 raft_consensus.cc:348] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.735389 2644 raft_consensus.cc:374] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.735699 2644 raft_consensus.cc:729] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.736385 2644 consensus_queue.cc:260] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.737593 2643 tablet_bootstrap.cc:492] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:47.789579 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:47.734431 2642 raft_consensus.cc:348] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.807750 2642 raft_consensus.cc:374] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.808568 2642 raft_consensus.cc:729] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
W20260806 03:50:47.809415 2538 leader_election.cc:341] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.738049 2644 ts_tablet_manager.cc:1428] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260806 03:50:47.820667 2817 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.822006 2817 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.820706 2886 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.823977 2886 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:47.832259 2534 leader_election.cc:341] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.833051 2534 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:47.738037 2643 ts_tablet_manager.cc:1397] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.090s user 0.028s sys 0.013s
I20260806 03:50:47.844991 2691 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:47.845916 2811 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.810340 2642 consensus_queue.cc:260] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.853497 2886 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:47.859364 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:47.861454 2534 leader_election.cc:341] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.865063 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.868244 2643 raft_consensus.cc:348] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.870203 2643 raft_consensus.cc:374] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:47.870640 2643 raft_consensus.cc:729] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:47.849917 2644 tablet_bootstrap.cc:492] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:47.872596 2818 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.897540 2899 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:47.855307 2642 ts_tablet_manager.cc:1428] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.128s user 0.006s sys 0.004s
I20260806 03:50:47.884893 2643 consensus_queue.cc:260] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.878795 2817 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:50:47.884510 2538 leader_election.cc:341] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.889895 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:47.895594 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.914793 2374 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:47.919190 2667 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.919906 2667 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.929949 2643 ts_tablet_manager.cc:1428] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.081s user 0.010s sys 0.004s
W20260806 03:50:47.925972 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 81516.5us
I20260806 03:50:47.944705 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:47.917053 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:47.948000 2667 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:47.925786 2534 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:47.955699 2886 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:47.956276 2886 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:47.956804 2886 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:47.958674 2644 tablet_bootstrap.cc:654] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:47.948753 2643 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:47.949203 2817 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:50:47.949375 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 64938.5us
W20260806 03:50:47.949987 2332 leader_election.cc:341] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:47.961612 2538 leader_election.cc:341] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:47.963615 2811 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:47.964269 2811 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:47.971208 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:47.950505 2642 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:47.972615 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:47.975417 2436 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:47.977968 2897 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:47.974462 2811 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:48.009325 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:48.032902 2886 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.035518 2886 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:48.039816 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:48.040779 2376 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:48.053763 2643 tablet_bootstrap.cc:654] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:48.054036 2482 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:48.056665 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:48.061801 2892 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:48.063009 2892 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.065737 2892 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:48.075533 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:48.075714 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:48.076498 2375 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:48.081770 2818 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:48.083461 2818 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.089313 2534 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:50:48.074318 2331 leader_election.cc:341] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:48.091373 2331 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:50:48.101786 2538 leader_election.cc:341] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:48.107995 2538 leader_election.cc:341] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:48.105441 2691 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:48.112838 2691 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:48.114326 2691 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:48.116851 2643 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:48.118835 2643 ts_tablet_manager.cc:1397] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.182s user 0.023s sys 0.008s
I20260806 03:50:48.120471 2920 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:50:48.131821 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 59781.3us
I20260806 03:50:48.132095 2981 consensus_queue.cc:1035] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
I20260806 03:50:48.137404 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:48.139364 2644 tablet_bootstrap.cc:492] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:48.158597 2644 ts_tablet_manager.cc:1397] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.345s user 0.018s sys 0.005s
I20260806 03:50:48.140301 2643 raft_consensus.cc:348] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.161218 2643 raft_consensus.cc:374] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:48.162003 2643 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:48.164644 2643 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:48.130061 2987 log.cc:924] Time spent T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260806 03:50:48.180658 2375 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:48.185122 2534 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:48.170786 2818 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:48.203336 2818 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:48.213325 2818 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:48.216351 2818 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:48.207259 2642 tablet_bootstrap.cc:654] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:48.244195 2644 raft_consensus.cc:348] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.273691 2644 raft_consensus.cc:374] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:48.274765 2644 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:48.277570 2643 ts_tablet_manager.cc:1428] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.158s user 0.016s sys 0.002s
I20260806 03:50:48.281993 2644 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.287712 2993 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
W20260806 03:50:48.304970 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 60994.7us
I20260806 03:50:48.293370 2376 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:48.340318 2643 tablet_bootstrap.cc:492] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:48.339028 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:48.403404 2644 ts_tablet_manager.cc:1428] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.244s user 0.031s sys 0.031s
W20260806 03:50:48.418573 2538 leader_election.cc:341] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:48.438129 2691 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.413293 2899 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:48.456846 2643 tablet_bootstrap.cc:654] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:48.459223 2644 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:48.479990 2899 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.491055 2899 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:48.527107 2691 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:48.519354 2534 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:48.532286 2892 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:48.533468 2892 raft_consensus.cc:686] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:48.522264 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:48.535782 2892 consensus_queue.cc:237] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.508554 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:48.588457 2818 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.600401 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:48.601930 2374 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:48.602087 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 345421us
I20260806 03:50:48.591387 2993 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:50:48.571938 2436 leader_election.cc:341] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:48.607681 2993 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:48.614331 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 44453.9us
I20260806 03:50:48.625885 2643 tablet_bootstrap.cc:492] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
W20260806 03:50:48.625133 2429 leader_election.cc:341] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:48.624346 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 47047.7us
I20260806 03:50:48.624828 2999 consensus_queue.cc:1035] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.052s
I20260806 03:50:48.680511 2643 ts_tablet_manager.cc:1397] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.357s user 0.024s sys 0.028s
I20260806 03:50:48.694070 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:48.680291 2429 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:48.679212 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:48.704421 2643 raft_consensus.cc:348] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.706975 2997 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:50:48.707000 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 82313.5us
I20260806 03:50:48.710820 2643 raft_consensus.cc:374] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:48.712364 2642 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:48.712646 2643 raft_consensus.cc:729] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:48.713338 2642 ts_tablet_manager.cc:1397] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.763s user 0.044s sys 0.047s
I20260806 03:50:48.676647 2818 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:48.713953 2643 consensus_queue.cc:260] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.717352 2379 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:48.718093 2534 leader_election.cc:341] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:48.716424 2993 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:48.733193 2643 ts_tablet_manager.cc:1428] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.045s user 0.006s sys 0.006s
I20260806 03:50:48.737154 2999 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:48.738312 2999 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.747393 2644 tablet_bootstrap.cc:654] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:48.748523 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:48.745440 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:48.750614 2483 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:48.753703 2643 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:48.797042 2999 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:48.832422 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:48.841763 2374 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:48.844385 2534 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:48.852452 2379 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:48.858240 2534 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:48.861555 2818 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:48.862673 2818 raft_consensus.cc:686] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:48.864590 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:48.860409 2642 raft_consensus.cc:348] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.867977 2642 raft_consensus.cc:374] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:48.868646 2642 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:48.861023 2996 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:48.866822 2818 consensus_queue.cc:237] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.873271 2996 raft_consensus.cc:686] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:48.875021 2996 consensus_queue.cc:237] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.879460 2644 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:48.880345 2644 ts_tablet_manager.cc:1397] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.449s user 0.031s sys 0.019s
I20260806 03:50:48.885854 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:48.884604 2538 leader_election.cc:341] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:48.902019 2642 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:48.902205 2584 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:48.908481 2483 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:48.922021 2642 ts_tablet_manager.cc:1428] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.208s user 0.005s sys 0.015s
I20260806 03:50:48.924479 2642 tablet_bootstrap.cc:492] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
W20260806 03:50:48.938247 2332 leader_election.cc:341] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:48.928102 2538 leader_election.cc:341] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:48.952669 2643 tablet_bootstrap.cc:654] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:48.953591 2538 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:48.961531 2232 catalog_manager.cc:5495] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:49.012995 2990 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 03:50:49.041803 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 113838us
I20260806 03:50:49.042506 2644 raft_consensus.cc:348] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.043500 2644 raft_consensus.cc:374] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.043829 2644 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.045614 2992 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:49.060490 2331 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.062812 2642 tablet_bootstrap.cc:654] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.048905 2644 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.080946 2644 ts_tablet_manager.cc:1428] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.200s user 0.007s sys 0.005s
I20260806 03:50:49.067498 3015 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:49.087085 2644 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:49.087496 3015 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:49.093825 3015 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.110083 2643 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:49.110692 2643 ts_tablet_manager.cc:1397] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.357s user 0.016s sys 0.018s
I20260806 03:50:49.148701 2644 tablet_bootstrap.cc:654] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.161836 2642 tablet_bootstrap.cc:492] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:49.162706 2642 ts_tablet_manager.cc:1397] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.239s user 0.016s sys 0.014s
I20260806 03:50:49.163467 2643 raft_consensus.cc:348] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.164847 2643 raft_consensus.cc:374] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.165218 2643 raft_consensus.cc:729] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.169253 2643 consensus_queue.cc:260] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:49.172729 2538 consensus_peers.cc:487] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:49.197847 3015 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.202114 2643 ts_tablet_manager.cc:1428] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.091s user 0.009s sys 0.005s
I20260806 03:50:49.203101 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:49.204624 2643 tablet_bootstrap.cc:492] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
W20260806 03:50:49.204676 2332 leader_election.cc:341] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.205384 2642 raft_consensus.cc:348] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.206218 2642 raft_consensus.cc:374] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.206516 2642 raft_consensus.cc:729] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.207242 2642 consensus_queue.cc:260] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.210704 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:49.211438 2584 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.211974 2644 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:49.212634 2644 ts_tablet_manager.cc:1397] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.130s user 0.013s sys 0.012s
I20260806 03:50:49.212957 3015 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:49.215523 2644 raft_consensus.cc:348] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.216637 2644 raft_consensus.cc:374] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.216969 2644 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.217828 2644 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.220409 2584 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:49.220419 2644 ts_tablet_manager.cc:1428] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 03:50:49.221727 2644 tablet_bootstrap.cc:492] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:49.221899 2331 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.219964 2643 tablet_bootstrap.cc:654] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.224820 3015 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
W20260806 03:50:49.224797 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:49.225312 2642 ts_tablet_manager.cc:1428] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.062s user 0.008s sys 0.000s
I20260806 03:50:49.225962 3015 raft_consensus.cc:686] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:49.226564 2642 tablet_bootstrap.cc:492] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:49.227049 3015 consensus_queue.cc:237] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.232862 2643 tablet_bootstrap.cc:492] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:49.233569 2643 ts_tablet_manager.cc:1397] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.000s
I20260806 03:50:49.234347 2644 tablet_bootstrap.cc:654] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.238070 2642 tablet_bootstrap.cc:654] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.237691 2643 raft_consensus.cc:348] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.239141 2643 raft_consensus.cc:374] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.252167 2643 raft_consensus.cc:729] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.253237 2643 consensus_queue.cc:260] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.262619 2643 ts_tablet_manager.cc:1428] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260806 03:50:49.263846 2643 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:49.267793 2691 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.268417 2691 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.270619 2643 tablet_bootstrap.cc:654] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.271726 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:49.272500 2374 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:49.275496 2534 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:49.276593 2990 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:49.276943 2990 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:49.277254 2990 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.276257 2236 catalog_manager.cc:5495] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:49.282582 2691 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.284451 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:49.288295 2990 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:49.289664 2538 leader_election.cc:341] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.291744 2990 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.292682 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:49.293396 2374 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.295150 2644 tablet_bootstrap.cc:492] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:49.295749 2644 ts_tablet_manager.cc:1397] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.074s user 0.009s sys 0.003s
I20260806 03:50:49.297667 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:50:49.299450 2538 leader_election.cc:341] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.301623 2644 raft_consensus.cc:348] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.302438 2644 raft_consensus.cc:374] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.302768 2644 raft_consensus.cc:729] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.301059 2232 catalog_manager.cc:5495] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:49.303715 2644 consensus_queue.cc:260] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.304666 2990 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.306049 2990 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.308912 2374 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:49.310534 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:49.311946 2379 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:49.313335 2534 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.314815 2642 tablet_bootstrap.cc:492] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:49.315377 2642 ts_tablet_manager.cc:1397] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.089s user 0.017s sys 0.011s
I20260806 03:50:49.316064 2993 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:49.316604 2993 raft_consensus.cc:686] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:49.317503 2993 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.307963 2232 catalog_manager.cc:5495] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:49.318619 2642 raft_consensus.cc:348] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.319474 2642 raft_consensus.cc:374] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.319770 2642 raft_consensus.cc:729] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.310571 2691 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.322439 2691 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.323513 2643 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:49.324873 2643 ts_tablet_manager.cc:1397] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.011s
I20260806 03:50:49.320549 2642 consensus_queue.cc:260] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.327382 2642 ts_tablet_manager.cc:1428] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.012s user 0.000s sys 0.007s
I20260806 03:50:49.323953 2534 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:49.327354 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:49.330622 2480 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:49.331261 2818 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:49.331717 2642 tablet_bootstrap.cc:492] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:49.332705 2538 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:49.334018 2892 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:49.334378 2892 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:49.334707 2892 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.331648 2818 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:49.335992 2818 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.334707 2643 raft_consensus.cc:348] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.339113 2643 raft_consensus.cc:374] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.339356 2643 raft_consensus.cc:729] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.341188 2644 ts_tablet_manager.cc:1428] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260806 03:50:49.328521 2691 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.343650 2643 consensus_queue.cc:260] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.348904 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:49.349820 2376 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:49.361011 2644 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:49.368187 2818 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.372484 2642 tablet_bootstrap.cc:654] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.375427 2990 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.376425 2643 ts_tablet_manager.cc:1428] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.051s user 0.007s sys 0.001s
I20260806 03:50:49.380234 2990 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.380901 2990 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.386296 2993 consensus_queue.cc:1035] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:49.391819 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:49.391582 2236 catalog_manager.cc:5495] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:49.392822 2375 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.394280 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:49.394944 2643 tablet_bootstrap.cc:492] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:49.394320 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:49.395689 2644 tablet_bootstrap.cc:654] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.395002 2480 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:49.397893 2990 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.400559 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:49.402127 2538 leader_election.cc:341] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.393584 2818 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.425035 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:49.427138 2534 leader_election.cc:341] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.427858 2534 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:49.430817 2993 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:49.432358 3015 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.412346 2892 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.434415 3015 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.437191 3015 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:49.448833 2892 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.468070 2481 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.468925 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:49.470916 2332 leader_election.cc:341] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.472448 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:49.473208 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:49.473594 2584 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:49.473883 2480 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.474659 2642 tablet_bootstrap.cc:492] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:49.475171 2642 ts_tablet_manager.cc:1397] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.144s user 0.032s sys 0.010s
I20260806 03:50:49.434060 2818 consensus_queue.cc:1035] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20260806 03:50:49.476759 2643 tablet_bootstrap.cc:654] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.482291 2644 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:49.482069 2892 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:49.482764 2644 ts_tablet_manager.cc:1397] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.122s user 0.010s sys 0.004s
I20260806 03:50:49.485072 2642 raft_consensus.cc:348] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.485854 2642 raft_consensus.cc:374] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.486159 2642 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.485078 2375 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:49.491362 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:49.492063 2377 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.496656 2899 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.497164 2899 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.498620 2534 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.499289 2899 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:49.489068 2642 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.500352 2691 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.500525 2992 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:49.500787 2691 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.501982 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:49.502501 2992 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:49.503029 2992 raft_consensus.cc:686] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:49.503882 2992 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.505687 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:49.503878 2691 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:50:49.508709 2538 leader_election.cc:341] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.507791 2331 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.518270 2836 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:49.518733 2836 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:49.518795 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:49.519132 2642 ts_tablet_manager.cc:1428] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.044s user 0.005s sys 0.004s
I20260806 03:50:49.519142 2836 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.520430 2642 tablet_bootstrap.cc:492] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:49.522979 2377 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:49.530984 2429 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:49.533524 2997 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:49.540057 2534 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:49.541213 2992 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:49.542341 2236 catalog_manager.cc:5495] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:49.543934 2992 raft_consensus.cc:686] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:49.544691 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:49.550655 2480 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:49.551154 2644 raft_consensus.cc:348] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.578985 2644 raft_consensus.cc:374] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.579296 2644 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.553579 2642 tablet_bootstrap.cc:654] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.567309 2836 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:49.576936 2534 leader_election.cc:341] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.577405 2992 consensus_queue.cc:237] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.583138 2836 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:49.585002 2534 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:49.579993 2644 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.585371 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:49.586278 2581 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.587153 2644 ts_tablet_manager.cc:1428] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.104s user 0.002s sys 0.009s
I20260806 03:50:49.588495 2644 tablet_bootstrap.cc:492] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:49.588719 3002 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.589181 3002 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.591190 3002 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:49.595571 2643 tablet_bootstrap.cc:492] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:49.592003 2990 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:49.596149 2643 ts_tablet_manager.cc:1397] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.201s user 0.023s sys 0.009s
I20260806 03:50:49.600984 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:49.603194 2332 leader_election.cc:341] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.605326 2818 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.606365 2818 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.607609 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:49.614578 2643 raft_consensus.cc:348] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.616217 2643 raft_consensus.cc:374] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.616573 2643 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.617460 2643 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.624082 2818 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.625463 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:49.625874 2332 leader_election.cc:341] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.626200 2584 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:49.627789 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.629541 3002 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:49.629424 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:49.629921 3002 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:49.630404 2644 tablet_bootstrap.cc:654] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.630322 3002 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.636917 2581 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:49.637490 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:49.638473 2331 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.639529 2981 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:49.640107 2981 raft_consensus.cc:686] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:49.641371 3002 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.641317 2981 consensus_queue.cc:237] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.642586 2374 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:50:49.647065 2538 leader_election.cc:341] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.650735 2534 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:49.652127 2993 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.661609 2643 ts_tablet_manager.cc:1428] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.065s user 0.006s sys 0.002s
I20260806 03:50:49.662885 2643 tablet_bootstrap.cc:492] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:49.662453 2232 catalog_manager.cc:5495] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:49.665148 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:50:49.666700 2332 leader_election.cc:341] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.672632 3002 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:49.674077 2481 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.675408 2642 tablet_bootstrap.cc:492] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:49.675971 2642 ts_tablet_manager.cc:1397] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.156s user 0.016s sys 0.031s
I20260806 03:50:49.687157 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:49.691601 2480 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.701900 2235 catalog_manager.cc:5495] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:49.703899 3002 consensus_queue.cc:1035] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:49.716651 2993 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:49.721062 2642 raft_consensus.cc:348] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.721868 2642 raft_consensus.cc:374] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.722126 2642 raft_consensus.cc:729] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.722718 2642 consensus_queue.cc:260] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.744946 2642 ts_tablet_manager.cc:1428] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.068s user 0.007s sys 0.000s
I20260806 03:50:49.761853 2581 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:49.770087 2642 tablet_bootstrap.cc:492] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
W20260806 03:50:49.770802 2538 consensus_peers.cc:487] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:49.775457 2644 tablet_bootstrap.cc:492] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:49.776026 2644 ts_tablet_manager.cc:1397] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.188s user 0.005s sys 0.007s
I20260806 03:50:49.778739 2981 consensus_queue.cc:1035] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:49.784902 2993 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.785857 2993 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.788663 2993 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:49.788673 2643 tablet_bootstrap.cc:654] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.791044 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.799461 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:49.802845 3002 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:49.803396 3002 raft_consensus.cc:686] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:49.803959 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:49.804224 3002 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:49.808859 2534 leader_election.cc:341] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.808873 2644 raft_consensus.cc:348] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.812085 2644 raft_consensus.cc:374] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.812762 2644 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.814693 2644 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:49.818131 2538 leader_election.cc:341] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.818660 2538 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:49.819465 2993 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:49.836259 2644 ts_tablet_manager.cc:1428] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.060s user 0.006s sys 0.002s
I20260806 03:50:49.837761 2644 tablet_bootstrap.cc:492] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:49.839594 2993 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:49.855548 2642 tablet_bootstrap.cc:654] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.856276 2643 tablet_bootstrap.cc:492] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:49.858160 2643 ts_tablet_manager.cc:1397] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.196s user 0.014s sys 0.004s
I20260806 03:50:49.861400 2643 raft_consensus.cc:348] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.862159 2643 raft_consensus.cc:374] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.863613 2643 raft_consensus.cc:729] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.864426 2643 consensus_queue.cc:260] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.866768 2643 ts_tablet_manager.cc:1428] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260806 03:50:49.869145 2643 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:49.875617 2233 catalog_manager.cc:5495] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:49.883461 2642 tablet_bootstrap.cc:492] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:49.884075 2642 ts_tablet_manager.cc:1397] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.122s user 0.015s sys 0.005s
I20260806 03:50:49.884588 2644 tablet_bootstrap.cc:654] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.886703 2642 raft_consensus.cc:348] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.887243 2642 raft_consensus.cc:374] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.887490 2642 raft_consensus.cc:729] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.890473 2643 tablet_bootstrap.cc:654] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.893707 2642 consensus_queue.cc:260] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.896286 2642 ts_tablet_manager.cc:1428] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260806 03:50:49.897535 2642 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:49.906838 2642 tablet_bootstrap.cc:654] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.926303 2644 tablet_bootstrap.cc:492] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:49.927800 2644 ts_tablet_manager.cc:1397] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.090s user 0.005s sys 0.013s
I20260806 03:50:49.933017 2644 raft_consensus.cc:348] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.936205 2644 raft_consensus.cc:374] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.936486 2644 raft_consensus.cc:729] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.937031 2644 consensus_queue.cc:260] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.940071 2644 ts_tablet_manager.cc:1428] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.012s user 0.008s sys 0.000s
I20260806 03:50:49.940598 2642 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:49.941165 2642 ts_tablet_manager.cc:1397] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.044s user 0.017s sys 0.020s
I20260806 03:50:49.942082 2644 tablet_bootstrap.cc:492] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:49.943988 2643 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:49.944531 2643 ts_tablet_manager.cc:1397] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.076s user 0.019s sys 0.018s
I20260806 03:50:49.951588 2643 raft_consensus.cc:348] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.952155 3015 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:49.952603 2643 raft_consensus.cc:374] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.952941 2643 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.952749 3015 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.953651 2643 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.955926 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:49.956115 2642 raft_consensus.cc:348] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.956902 2642 raft_consensus.cc:374] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:49.957190 2642 raft_consensus.cc:729] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:49.956446 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:49.958019 2578 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:49.958133 2644 tablet_bootstrap.cc:654] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.957871 2642 consensus_queue.cc:260] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.962105 2643 ts_tablet_manager.cc:1428] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
W20260806 03:50:49.959969 2332 leader_election.cc:341] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:49.963586 2643 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:49.960340 2642 ts_tablet_manager.cc:1428] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.019s user 0.008s sys 0.000s
I20260806 03:50:49.964560 2892 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:49.971518 2331 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:49.972591 3002 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:49.972986 3002 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:49.973307 3002 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:49.977191 3015 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:49.980734 2642 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:49.980499 2892 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:49.985674 3002 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:49.987257 2642 tablet_bootstrap.cc:654] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.977722 2643 tablet_bootstrap.cc:654] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:49.996376 2892 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:50.012686 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:50:50.015110 2332 leader_election.cc:341] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.019407 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:50.020167 2578 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.011322 3002 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.045243 2480 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.052204 2478 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.064636 2899 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.064905 2644 tablet_bootstrap.cc:492] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:50.065280 2899 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.066329 2644 ts_tablet_manager.cc:1397] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.124s user 0.030s sys 0.000s
I20260806 03:50:50.069916 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:50.071746 2899 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.071605 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:50.075922 2643 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:50.076464 2643 ts_tablet_manager.cc:1397] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.113s user 0.014s sys 0.008s
I20260806 03:50:50.076397 2644 raft_consensus.cc:348] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.077046 2644 raft_consensus.cc:374] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.072391 2581 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:50:50.077225 2644 raft_consensus.cc:729] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:50.082901 2436 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:50.083333 2644 consensus_queue.cc:260] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.085608 2899 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.089769 2643 raft_consensus.cc:348] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.090551 2643 raft_consensus.cc:374] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.090821 2643 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:50.091526 2643 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.069983 2578 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:50.098647 2644 ts_tablet_manager.cc:1428] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260806 03:50:50.099735 2644 tablet_bootstrap.cc:492] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:50.102174 2331 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:50.103982 3015 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:50.104481 3015 raft_consensus.cc:686] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:50.105336 3015 consensus_queue.cc:237] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.110811 2899 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:50.111840 2643 ts_tablet_manager.cc:1428] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.035s user 0.006s sys 0.000s
I20260806 03:50:50.113641 2981 consensus_queue.cc:1035] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:50.114586 2892 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:50.120543 2644 tablet_bootstrap.cc:654] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:50.130568 2643 tablet_bootstrap.cc:492] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:50.141911 2642 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:50.142527 2642 ts_tablet_manager.cc:1397] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.004s
I20260806 03:50:50.159111 3015 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.159739 3015 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.160063 2232 catalog_manager.cc:5495] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:50.162633 3015 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.163648 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:50.165403 2332 leader_election.cc:341] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.167445 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:50.167475 2642 raft_consensus.cc:348] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.168210 2581 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:50.168516 2642 raft_consensus.cc:374] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.168870 2642 raft_consensus.cc:729] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:50.169488 2331 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:50.169565 2642 consensus_queue.cc:260] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.170421 2644 tablet_bootstrap.cc:492] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:50.170725 2836 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:50.170943 2644 ts_tablet_manager.cc:1397] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.071s user 0.014s sys 0.012s
I20260806 03:50:50.171169 2836 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:50.171590 2836 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.174036 2644 raft_consensus.cc:348] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.174906 2644 raft_consensus.cc:374] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.175253 2644 raft_consensus.cc:729] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:50.177047 2643 tablet_bootstrap.cc:654] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:50.176378 2644 consensus_queue.cc:260] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.178562 2642 ts_tablet_manager.cc:1428] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.036s user 0.010s sys 0.000s
I20260806 03:50:50.186038 2644 ts_tablet_manager.cc:1428] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20260806 03:50:50.189954 2644 tablet_bootstrap.cc:492] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:50.193735 2642 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:50.203833 2836 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.214031 3002 consensus_queue.cc:1035] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:50.226367 2922 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.226966 2922 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.227900 2643 tablet_bootstrap.cc:492] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:50.228497 2643 ts_tablet_manager.cc:1397] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.099s user 0.015s sys 0.006s
I20260806 03:50:50.229339 2922 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.231087 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:50.231429 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:50.235966 2332 leader_election.cc:341] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:50.236693 2332 leader_election.cc:341] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.241386 2644 tablet_bootstrap.cc:654] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:50.239853 2643 raft_consensus.cc:348] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.242501 2643 raft_consensus.cc:374] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.242769 2643 raft_consensus.cc:729] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:50.243392 2643 consensus_queue.cc:260] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.250409 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:50.250813 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:50.251513 2584 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.251250 2578 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:50.243983 2836 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.271023 2899 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.271723 2899 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.276739 2643 ts_tablet_manager.cc:1428] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.048s user 0.007s sys 0.000s
I20260806 03:50:50.277810 2899 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.280556 2993 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.286367 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:50.286494 2993 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.290215 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:50.291637 2643 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
W20260806 03:50:50.303297 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:50.292146 2981 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.306113 2981 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.306859 2993 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:50.309368 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:50.310395 2981 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:50.310988 2332 leader_election.cc:341] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:50.312965 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 62444.1us
I20260806 03:50:50.315131 2331 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:50.316166 3002 raft_consensus.cc:2781] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:50.316556 3002 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:50.316920 3002 raft_consensus.cc:3037] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.320529 2436 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:50.322572 2642 tablet_bootstrap.cc:654] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:50.323786 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:50.324157 2899 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:50.325917 2584 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:50.340746 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:50.342054 2331 leader_election.cc:341] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.342563 2331 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:50.343838 2331 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:50.348223 3093 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:50.350262 3093 raft_consensus.cc:686] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:50.346832 2981 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:50:50.351296 2538 leader_election.cc:341] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.351186 3093 consensus_queue.cc:237] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.358462 2478 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.366218 2981 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:50.392879 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:50.396436 2643 tablet_bootstrap.cc:654] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:50.404181 3091 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 03:50:50.406935 3002 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.410130 3002 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.417357 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:50.418318 2583 raft_consensus.cc:3037] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.462868 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:50.463429 2644 tablet_bootstrap.cc:492] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:50.464002 2644 ts_tablet_manager.cc:1397] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.274s user 0.031s sys 0.038s
I20260806 03:50:50.465078 2642 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:50.465749 2642 ts_tablet_manager.cc:1397] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.282s user 0.035s sys 0.010s
I20260806 03:50:50.473716 2642 raft_consensus.cc:348] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.474668 2642 raft_consensus.cc:374] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.475032 2642 raft_consensus.cc:729] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
W20260806 03:50:50.475966 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 61786.5us
I20260806 03:50:50.475857 2642 consensus_queue.cc:260] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:50.476698 2534 leader_election.cc:341] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.477506 2534 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:50.478298 2642 ts_tablet_manager.cc:1428] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20260806 03:50:50.479573 2691 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:50.477541 2583 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:50.480883 2644 raft_consensus.cc:348] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.481588 2644 raft_consensus.cc:374] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.481818 2644 raft_consensus.cc:729] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:50.482491 2644 consensus_queue.cc:260] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:50.474205 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 61787.2us
W20260806 03:50:50.486982 2332 leader_election.cc:341] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.489987 2331 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:50.491189 3002 raft_consensus.cc:2781] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:50.500270 2642 tablet_bootstrap.cc:492] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:50.501287 3002 raft_consensus.cc:686] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:50.502452 2644 ts_tablet_manager.cc:1428] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.035s user 0.004s sys 0.002s
I20260806 03:50:50.503672 2644 tablet_bootstrap.cc:492] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:50.497802 2232 catalog_manager.cc:5495] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:50.515990 2981 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.519630 2981 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.522173 2981 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.527565 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:50.529264 2332 leader_election.cc:341] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.523439 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:50.543637 2583 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:50.546051 2331 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:50.548179 3091 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:50.548817 3091 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:50.502665 3002 consensus_queue.cc:237] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.549326 3091 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:50.590847 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 76990.2us
I20260806 03:50:50.608142 2643 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:50.608754 2643 ts_tablet_manager.cc:1397] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.317s user 0.014s sys 0.033s
I20260806 03:50:50.623867 2643 raft_consensus.cc:348] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.624846 2643 raft_consensus.cc:374] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.625144 2643 raft_consensus.cc:729] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:50.626852 2643 consensus_queue.cc:260] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.637351 2818 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.653126 2818 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.658584 2643 ts_tablet_manager.cc:1428] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.049s user 0.008s sys 0.000s
I20260806 03:50:50.664459 2643 tablet_bootstrap.cc:492] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:50.667850 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:50.669006 2482 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:50.660183 2818 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:50.673246 2818 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.675283 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:50.679309 2993 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:50.688236 3110 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:50.688798 3110 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.690601 2993 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:50.697916 2993 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.698951 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:50.700608 2481 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.674765 2818 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.714154 2642 tablet_bootstrap.cc:654] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:50.729166 3089 consensus_queue.cc:1035] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:50.733341 2818 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:50.734856 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:50.751113 2644 tablet_bootstrap.cc:654] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:50.711764 3091 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.764075 3091 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.766722 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:50.769574 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:50.769476 2993 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:50.771134 2332 leader_election.cc:341] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.772485 2436 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:50.773375 2997 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:50:50.777613 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 76060.5us
I20260806 03:50:50.781198 3015 consensus_queue.cc:1035] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:50.781490 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:50.771082 3110 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:50.790300 2643 tablet_bootstrap.cc:654] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
W20260806 03:50:50.784401 2534 leader_election.cc:341] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.776527 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:50:50.796748 2534 leader_election.cc:341] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:50.796757 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 94581.6us
I20260806 03:50:50.799050 2993 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:50.800523 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:50.802311 2583 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:50.812423 2534 leader_election.cc:341] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.812737 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:50.813694 2479 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:50.825686 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:50.861531 2642 tablet_bootstrap.cc:492] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:50.862142 2642 ts_tablet_manager.cc:1397] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.363s user 0.010s sys 0.018s
W20260806 03:50:50.862023 2538 leader_election.cc:341] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:50.862951 2538 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:50.864284 3115 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:50.863816 2479 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:50.876953 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:50.878314 2691 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:50.878935 2691 raft_consensus.cc:686] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:50.880041 2691 consensus_queue.cc:237] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.880535 2642 raft_consensus.cc:348] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.881580 2642 raft_consensus.cc:374] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:50.872282 3015 consensus_queue.cc:1035] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260806 03:50:50.881937 2642 raft_consensus.cc:729] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:50.895210 2642 consensus_queue.cc:260] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:50.886466 2583 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
W20260806 03:50:51.093091 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 198830us
I20260806 03:50:51.096582 2644 tablet_bootstrap.cc:492] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:51.097169 2644 ts_tablet_manager.cc:1397] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.594s user 0.019s sys 0.052s
I20260806 03:50:51.099807 2642 ts_tablet_manager.cc:1428] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.237s user 0.007s sys 0.011s
I20260806 03:50:51.100996 2642 tablet_bootstrap.cc:492] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:51.108729 2331 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:51.115533 2922 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:51.123150 2922 raft_consensus.cc:686] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:51.124178 2922 consensus_queue.cc:237] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.138393 2643 tablet_bootstrap.cc:492] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:51.139197 2643 ts_tablet_manager.cc:1397] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.476s user 0.019s sys 0.041s
I20260806 03:50:51.157192 2642 tablet_bootstrap.cc:654] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:51.171622 2643 raft_consensus.cc:348] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.172598 2643 raft_consensus.cc:374] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:51.172333 2644 raft_consensus.cc:348] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.172888 2643 raft_consensus.cc:729] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:51.173081 2644 raft_consensus.cc:374] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:51.173405 2644 raft_consensus.cc:729] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:51.173709 2643 consensus_queue.cc:260] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.178656 2233 catalog_manager.cc:5495] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:51.187462 2643 ts_tablet_manager.cc:1428] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.048s user 0.008s sys 0.001s
I20260806 03:50:51.219849 2484 raft_consensus.cc:3037] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:51.241266 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 70486.2us
I20260806 03:50:51.371222 2643 tablet_bootstrap.cc:492] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
W20260806 03:50:51.373662 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 54877.2us
I20260806 03:50:51.421126 2642 tablet_bootstrap.cc:492] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:51.421751 2642 ts_tablet_manager.cc:1397] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.321s user 0.024s sys 0.016s
W20260806 03:50:51.413794 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 152234us
I20260806 03:50:51.478556 2642 raft_consensus.cc:348] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:51.485939 2332 consensus_peers.cc:487] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:50:51.520344 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 260020us
W20260806 03:50:51.493073 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:51.500206 3110 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:51.523145 3110 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.528242 3110 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:51.499096 2643 tablet_bootstrap.cc:654] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:51.479519 2642 raft_consensus.cc:374] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:51.530174 2642 raft_consensus.cc:729] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
W20260806 03:50:51.508102 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 106446us
I20260806 03:50:51.533169 2892 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:51.533900 2892 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:51.516571 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 127524us
I20260806 03:50:51.531036 2642 consensus_queue.cc:260] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.501811 2236 catalog_manager.cc:5495] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:51.511493 2644 consensus_queue.cc:260] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.560094 2644 ts_tablet_manager.cc:1428] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.463s user 0.009s sys 0.003s
I20260806 03:50:51.566155 2892 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:51.566917 2644 tablet_bootstrap.cc:492] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:51.587766 2642 ts_tablet_manager.cc:1428] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.166s user 0.005s sys 0.005s
I20260806 03:50:51.589314 2642 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:51.609046 3144 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:51.609669 3144 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.624956 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:51.636793 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:51.637548 2576 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:51.627719 3145 consensus_queue.cc:1035] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.114s
I20260806 03:50:51.634409 3144 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:51.645541 2331 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:51.640915 2376 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:51.648293 3140 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:51.644503 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:51.648695 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:51.648906 2583 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:51.652396 3112 consensus_queue.cc:1035] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 03:50:51.648661 3140 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:51.657168 3140 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:51.661808 3151 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:51.671209 2436 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:51.672396 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:51.675457 3159 raft_consensus.cc:2781] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:51.675886 3159 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:51.676265 3159 raft_consensus.cc:3037] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:51.664876 3151 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:51.680190 3151 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:51.681128 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:51.714875 2534 consensus_peers.cc:487] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c -> Peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Couldn't send request to peer dff94100ffc5485f92cbe6a85addf0b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260806 03:50:51.787456 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 72732.5us
I20260806 03:50:51.691210 2236 catalog_manager.cc:5495] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:51.731920 2642 tablet_bootstrap.cc:654] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:51.791462 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:51.795174 2644 tablet_bootstrap.cc:654] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:51.787807 2643 tablet_bootstrap.cc:492] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:51.802639 2643 ts_tablet_manager.cc:1397] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.478s user 0.020s sys 0.021s
I20260806 03:50:51.818243 2643 raft_consensus.cc:348] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.819121 2643 raft_consensus.cc:374] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:51.819401 2643 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:51.822585 2643 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.776124 3159 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.825021 2481 raft_consensus.cc:1226] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260806 03:50:51.825845 3159 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:51.826514 2643 ts_tablet_manager.cc:1428] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.024s user 0.002s sys 0.005s
I20260806 03:50:51.827745 2643 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:51.828390 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:51.832000 3151 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.835721 2644 tablet_bootstrap.cc:492] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:51.836261 2644 ts_tablet_manager.cc:1397] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.270s user 0.014s sys 0.014s
I20260806 03:50:51.838987 2644 raft_consensus.cc:348] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.839723 2644 raft_consensus.cc:374] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:51.840008 2644 raft_consensus.cc:729] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:51.845379 3151 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:51.848308 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:51.841739 2644 consensus_queue.cc:260] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.855763 2644 ts_tablet_manager.cc:1428] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260806 03:50:51.857262 2644 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:51.868388 2643 tablet_bootstrap.cc:654] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:51.889479 2643 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
W20260806 03:50:51.889794 2332 leader_election.cc:341] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:51.890038 2643 ts_tablet_manager.cc:1397] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.007s
I20260806 03:50:51.890687 3140 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:51.892001 2332 consensus_peers.cc:487] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:51.892504 2373 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:50:51.894282 3089 consensus_queue.cc:1046] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:51.895033 3144 consensus_queue.cc:1035] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:51.896149 2643 raft_consensus.cc:348] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.896876 2643 raft_consensus.cc:374] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:51.897120 2643 raft_consensus.cc:729] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:51.897701 2643 consensus_queue.cc:260] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:51.903127 3089 raft_consensus.cc:3032] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:51.903771 3089 raft_consensus.cc:729] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:51.905514 2643 ts_tablet_manager.cc:1428] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260806 03:50:51.906620 2643 tablet_bootstrap.cc:492] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:51.909936 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:51.904629 3089 consensus_queue.cc:260] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:51.917090 2538 leader_election.cc:341] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:51.909292 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:51.918344 3089 raft_consensus.cc:3037] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:51.931331 3153 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260806 03:50:51.940604 2538 leader_election.cc:341] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:51.942162 2538 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:51.944636 3152 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:50:51.955752 2642 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:51.959199 2642 ts_tablet_manager.cc:1397] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.370s user 0.050s sys 0.056s
I20260806 03:50:51.968247 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:50:52.003580 2332 leader_election.cc:341] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.004604 2691 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.005044 3148 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.005491 2691 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.005715 3148 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.013010 2642 raft_consensus.cc:348] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.013882 2642 raft_consensus.cc:374] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.014200 2642 raft_consensus.cc:729] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
W20260806 03:50:52.013953 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 43677.6us
I20260806 03:50:52.016213 3140 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.017025 2691 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:52.019634 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:52.021313 2643 tablet_bootstrap.cc:654] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.014938 2642 consensus_queue.cc:260] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.019634 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:52.023118 2376 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:52.024632 3148 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.025466 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:52.026013 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:52.026628 2583 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:52.027006 2579 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:52.028469 2436 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:52.028788 2534 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:50:52.030017 2538 leader_election.cc:341] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.031350 2372 raft_consensus.cc:2396] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260806 03:50:52.031854 3148 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:52.032528 3148 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:52.033113 3152 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:52.034089 3152 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:52.033032 3148 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.034556 3152 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.033351 2331 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.046043 3089 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:50:52.048111 2642 ts_tablet_manager.cc:1428] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.088s user 0.008s sys 0.001s
I20260806 03:50:52.058573 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:52.070793 2644 tablet_bootstrap.cc:654] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.072557 2642 tablet_bootstrap.cc:492] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:52.075354 2429 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.077327 3159 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 03:50:52.078109 2429 leader_election.cc:341] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.125880 3162 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.137440 3162 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.157786 3162 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:52.158429 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:52.158824 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:52.159825 2534 leader_election.cc:341] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.161842 2643 tablet_bootstrap.cc:492] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:52.162384 2643 ts_tablet_manager.cc:1397] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.256s user 0.013s sys 0.019s
I20260806 03:50:52.165894 2643 raft_consensus.cc:348] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.166651 2643 raft_consensus.cc:374] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.166934 2643 raft_consensus.cc:729] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.167658 2643 consensus_queue.cc:260] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:52.169533 2538 leader_election.cc:341] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.170001 2538 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:52.171636 3162 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:52.173050 3140 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.173550 3140 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.177002 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:52.177217 3140 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:52.178872 2331 leader_election.cc:341] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.180258 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:52.180616 3148 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.182570 2481 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.185067 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:52.185721 2584 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.186128 2642 tablet_bootstrap.cc:654] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.179893 2643 ts_tablet_manager.cc:1428] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
W20260806 03:50:52.188433 2332 leader_election.cc:341] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.189391 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.192891 3148 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.193744 2643 tablet_bootstrap.cc:492] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:52.197031 3152 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.202981 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:52.203688 2372 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.205475 2584 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:52.205232 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:52.200822 3015 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:52.207120 2436 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:50:52.209775 2429 leader_election.cc:341] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.208920 3160 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:52.210943 3160 raft_consensus.cc:686] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:52.211767 3160 consensus_queue.cc:237] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.217612 3152 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:52.223378 2644 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:52.224128 2644 ts_tablet_manager.cc:1397] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.367s user 0.024s sys 0.016s
I20260806 03:50:52.224807 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:52.236907 2644 raft_consensus.cc:348] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.240752 2644 raft_consensus.cc:374] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.241276 2644 raft_consensus.cc:729] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
W20260806 03:50:52.247458 2538 leader_election.cc:341] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.242242 2644 consensus_queue.cc:260] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.257017 2643 tablet_bootstrap.cc:654] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.259853 2644 ts_tablet_manager.cc:1428] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.035s user 0.008s sys 0.002s
I20260806 03:50:52.261359 2644 tablet_bootstrap.cc:492] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:52.291023 2233 catalog_manager.cc:5495] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:52.310770 2644 tablet_bootstrap.cc:654] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.273654 2372 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:52.318143 2534 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:52.318958 3152 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:52.319442 3152 raft_consensus.cc:686] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:52.320289 3152 consensus_queue.cc:237] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.330783 3089 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.331352 3089 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.333995 3089 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.336824 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:52.346565 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:52.347739 2584 raft_consensus.cc:2379] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 1: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:50:52.349037 2331 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.350617 3015 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.381449 2644 tablet_bootstrap.cc:492] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:52.382102 2644 ts_tablet_manager.cc:1397] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.121s user 0.012s sys 0.009s
I20260806 03:50:52.397620 3015 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:52.402787 2644 raft_consensus.cc:348] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.403615 2644 raft_consensus.cc:374] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.403882 2644 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.399292 2642 tablet_bootstrap.cc:492] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:52.404700 2642 ts_tablet_manager.cc:1397] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.332s user 0.037s sys 0.091s
I20260806 03:50:52.404986 2644 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.408215 2643 tablet_bootstrap.cc:492] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:52.408746 2643 ts_tablet_manager.cc:1397] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.215s user 0.023s sys 0.008s
I20260806 03:50:52.410488 2644 ts_tablet_manager.cc:1428] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.028s user 0.009s sys 0.000s
I20260806 03:50:52.415464 2643 raft_consensus.cc:348] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.410236 2236 catalog_manager.cc:5495] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:52.422158 2643 raft_consensus.cc:374] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.422479 2643 raft_consensus.cc:729] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.423154 2643 consensus_queue.cc:260] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.420127 2642 raft_consensus.cc:348] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.424175 2642 raft_consensus.cc:374] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.424443 2642 raft_consensus.cc:729] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.425109 2642 consensus_queue.cc:260] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.425690 2643 ts_tablet_manager.cc:1428] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.017s user 0.004s sys 0.002s
I20260806 03:50:52.426748 2643 tablet_bootstrap.cc:492] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:52.427626 2644 tablet_bootstrap.cc:492] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:52.428223 3089 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.428704 3089 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.432109 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:52.432684 2642 ts_tablet_manager.cc:1428] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.028s user 0.004s sys 0.004s
I20260806 03:50:52.433008 2643 tablet_bootstrap.cc:654] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.433845 2642 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
W20260806 03:50:52.434131 2332 leader_election.cc:341] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.436837 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:50:52.438230 2331 leader_election.cc:341] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.438635 2331 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.439431 3145 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:52.443207 2644 tablet_bootstrap.cc:654] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.448145 2642 tablet_bootstrap.cc:654] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.460199 2642 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:52.460878 2642 ts_tablet_manager.cc:1397] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.003s
I20260806 03:50:52.464320 2642 raft_consensus.cc:348] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.465332 2642 raft_consensus.cc:374] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.465652 2642 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.466516 2642 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.468713 3089 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.478367 2642 ts_tablet_manager.cc:1428] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.017s user 0.008s sys 0.002s
I20260806 03:50:52.481069 2644 tablet_bootstrap.cc:492] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:52.481473 2642 tablet_bootstrap.cc:492] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:52.481652 2644 ts_tablet_manager.cc:1397] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.054s user 0.021s sys 0.004s
I20260806 03:50:52.482784 2643 tablet_bootstrap.cc:492] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:52.483273 2643 ts_tablet_manager.cc:1397] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.033s
I20260806 03:50:52.485159 2644 raft_consensus.cc:348] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.486007 2644 raft_consensus.cc:374] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.485826 2643 raft_consensus.cc:348] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.486315 2644 raft_consensus.cc:729] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.486555 2643 raft_consensus.cc:374] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.486829 2643 raft_consensus.cc:729] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.487016 2644 consensus_queue.cc:260] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.487515 2643 consensus_queue.cc:260] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.488160 2642 tablet_bootstrap.cc:654] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.489598 2644 ts_tablet_manager.cc:1428] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260806 03:50:52.490679 2644 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:52.499306 2644 tablet_bootstrap.cc:654] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.504746 3160 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.505403 3160 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.519734 2642 tablet_bootstrap.cc:492] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:52.520069 2644 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:52.520423 2642 ts_tablet_manager.cc:1397] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.022s
I20260806 03:50:52.520617 2644 ts_tablet_manager.cc:1397] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.030s user 0.020s sys 0.008s
I20260806 03:50:52.521613 3160 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.523396 2642 raft_consensus.cc:348] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.524238 2642 raft_consensus.cc:374] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.524518 2642 raft_consensus.cc:729] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.524124 2644 raft_consensus.cc:348] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.524858 2644 raft_consensus.cc:374] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.525126 2644 raft_consensus.cc:729] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.525256 2642 consensus_queue.cc:260] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.525746 2644 consensus_queue.cc:260] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.528358 2643 ts_tablet_manager.cc:1428] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.045s user 0.003s sys 0.003s
I20260806 03:50:52.529117 2644 ts_tablet_manager.cc:1428] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260806 03:50:52.529429 2643 tablet_bootstrap.cc:492] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:52.530087 2644 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:52.530184 2818 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.530642 2818 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.532802 2818 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:52.534689 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:52.535455 2376 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:52.536720 2643 tablet_bootstrap.cc:654] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.536993 2642 ts_tablet_manager.cc:1428] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20260806 03:50:52.538117 2642 tablet_bootstrap.cc:492] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:52.538573 2534 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:52.540393 3162 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:52.540730 3162 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:52.541064 3162 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.545159 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:52.545877 2481 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:52.546231 2642 tablet_bootstrap.cc:654] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.551671 2644 tablet_bootstrap.cc:654] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.552217 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:52.571900 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:52.578395 2643 tablet_bootstrap.cc:492] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:52.578445 2429 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.579316 2643 ts_tablet_manager.cc:1397] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.050s user 0.008s sys 0.032s
I20260806 03:50:52.580116 3160 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:52.583037 2643 raft_consensus.cc:348] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.584007 2643 raft_consensus.cc:374] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.588196 2644 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:52.592180 2643 raft_consensus.cc:729] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.592751 2644 ts_tablet_manager.cc:1397] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.063s user 0.015s sys 0.013s
I20260806 03:50:52.593016 2643 consensus_queue.cc:260] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.595372 2643 ts_tablet_manager.cc:1428] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.016s user 0.006s sys 0.003s
I20260806 03:50:52.586858 2481 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.595489 2644 raft_consensus.cc:348] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.596479 2644 raft_consensus.cc:374] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.596741 2644 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.597415 2644 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.599627 2644 ts_tablet_manager.cc:1428] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260806 03:50:52.600651 2644 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:52.606923 2644 tablet_bootstrap.cc:654] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.609047 2643 tablet_bootstrap.cc:492] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:52.612247 3162 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.616194 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:52.617060 2376 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.619504 2644 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:52.620071 2644 ts_tablet_manager.cc:1397] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.012s
I20260806 03:50:52.621366 2818 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.621856 2818 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.622676 2644 raft_consensus.cc:348] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.623462 2644 raft_consensus.cc:374] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.623719 2644 raft_consensus.cc:729] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.624444 3160 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:52.624405 2644 consensus_queue.cc:260] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.628715 3152 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.629216 3152 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.630961 2643 tablet_bootstrap.cc:654] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.631618 3152 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:52.632893 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:52.634384 2538 leader_election.cc:341] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.636061 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:52.636646 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:52.636791 2372 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:52.638240 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:52.639714 2538 leader_election.cc:341] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.640347 2818 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:52.641439 2643 tablet_bootstrap.cc:492] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:52.641281 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:52.641963 2643 ts_tablet_manager.cc:1397] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.005s
I20260806 03:50:52.644749 2534 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:52.646008 2644 ts_tablet_manager.cc:1428] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260806 03:50:52.646503 3152 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:52.646987 2644 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
W20260806 03:50:52.650115 2534 leader_election.cc:341] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.645327 2643 raft_consensus.cc:348] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.650588 2534 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:52.650825 2643 raft_consensus.cc:374] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.651132 2643 raft_consensus.cc:729] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.651324 3151 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:52.651769 2643 consensus_queue.cc:260] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.654477 2643 ts_tablet_manager.cc:1428] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.012s user 0.005s sys 0.002s
I20260806 03:50:52.655462 2643 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:52.646898 3152 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:52.658471 3152 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.674655 2644 tablet_bootstrap.cc:654] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.686936 2644 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:52.687446 2644 ts_tablet_manager.cc:1397] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.007s
I20260806 03:50:52.696501 2644 raft_consensus.cc:348] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.697254 2644 raft_consensus.cc:374] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.697531 2644 raft_consensus.cc:729] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.697691 3160 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.698194 3160 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.700297 3160 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.698222 2644 consensus_queue.cc:260] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.704116 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:52.706142 2429 leader_election.cc:341] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.706584 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:52.707675 2583 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:52.709353 2436 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:52.710433 3148 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:52.710816 3148 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:52.711171 3148 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.713701 3159 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.714219 3159 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.718578 3159 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.719185 3148 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.721645 3160 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.722200 3160 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.723323 3148 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.725682 3160 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.728382 2644 ts_tablet_manager.cc:1428] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.041s user 0.005s sys 0.004s
I20260806 03:50:52.728231 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:52.729022 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:52.729707 2374 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:52.730738 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:52.731804 2373 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:50:52.733755 2429 leader_election.cc:341] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.735199 2429 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:52.736502 3160 raft_consensus.cc:2781] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:52.737175 3160 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:52.737596 3160 raft_consensus.cc:3037] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.736366 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:52.742067 2583 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:52.751917 3160 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.754746 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:52.759989 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:52.760942 2373 raft_consensus.cc:3037] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.764861 3162 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:52.767302 2643 tablet_bootstrap.cc:654] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.769109 3159 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:52.770970 2644 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:52.773202 3160 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.774051 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:52.774961 2579 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.776008 2436 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.784641 2376 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
W20260806 03:50:52.787628 2436 leader_election.cc:341] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.798146 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:50:52.800635 2436 leader_election.cc:341] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:52.811683 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 82601.8us
I20260806 03:50:52.814823 2644 tablet_bootstrap.cc:654] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:52.816157 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:52.816859 2482 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.797021 3148 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:52.833813 2534 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:52.838302 2818 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:52.839052 2818 raft_consensus.cc:686] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:52.840236 2818 consensus_queue.cc:237] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.763825 3152 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.855508 2642 tablet_bootstrap.cc:492] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:52.856616 2642 ts_tablet_manager.cc:1397] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.319s user 0.054s sys 0.085s
I20260806 03:50:52.844321 3015 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.860359 3015 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.861804 2482 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:52.862601 3015 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.864459 2642 raft_consensus.cc:348] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.865381 2642 raft_consensus.cc:374] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:52.865682 2642 raft_consensus.cc:729] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:52.867516 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:52.867529 2642 consensus_queue.cc:260] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.851735 2373 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:52.867751 3089 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.890522 2642 ts_tablet_manager.cc:1428] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.034s user 0.008s sys 0.000s
I20260806 03:50:52.896994 2429 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.898699 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:52.900429 2642 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:52.900576 2331 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:52.903416 3145 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:52.892982 3089 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.912894 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:52.915910 2332 leader_election.cc:341] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:52.914165 3218 raft_consensus.cc:2781] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:52.933722 3162 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:52.934525 3162 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:52.939751 3152 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:52.940210 2579 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
W20260806 03:50:52.949534 2538 consensus_peers.cc:487] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:52.925343 3089 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:52.982033 2232 catalog_manager.cc:5495] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:52.949110 3218 raft_consensus.cc:686] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:50:53.015786 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 105428us
I20260806 03:50:53.044284 2643 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:53.063174 2643 ts_tablet_manager.cc:1397] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.408s user 0.011s sys 0.009s
I20260806 03:50:53.068806 2436 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:53.044039 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:53.071650 3148 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:53.092938 2642 tablet_bootstrap.cc:654] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:53.080193 3159 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:53.108891 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:53.117326 2579 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:53.111284 3159 raft_consensus.cc:686] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:53.119715 3159 consensus_queue.cc:237] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.118656 2644 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:53.123878 2644 ts_tablet_manager.cc:1397] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.353s user 0.018s sys 0.010s
I20260806 03:50:53.089263 3218 consensus_queue.cc:237] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.141677 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:53.142544 2379 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:53.161353 2379 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:53.134316 2643 raft_consensus.cc:348] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.176055 2643 raft_consensus.cc:374] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:53.176535 2643 raft_consensus.cc:729] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:53.177320 2643 consensus_queue.cc:260] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.189138 2331 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:53.193356 2478 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:53.209496 2644 raft_consensus.cc:348] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.213799 2644 raft_consensus.cc:374] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:53.214360 2644 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:53.217067 2643 ts_tablet_manager.cc:1428] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.153s user 0.012s sys 0.001s
I20260806 03:50:53.215377 2644 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.225534 2534 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:53.240959 2818 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260806 03:50:53.212249 3145 raft_consensus.cc:2781] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:53.259049 2643 tablet_bootstrap.cc:492] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:53.260712 3162 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:53.263290 2644 ts_tablet_manager.cc:1428] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.130s user 0.009s sys 0.001s
I20260806 03:50:53.286616 2478 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:53.283427 3162 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:53.257596 3152 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:53.304304 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:53.265097 3145 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:50:53.307893 2538 leader_election.cc:341] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:53.311688 2379 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:53.311896 3152 raft_consensus.cc:686] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:53.315630 3152 consensus_queue.cc:237] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.317142 3210 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:53.317420 2644 tablet_bootstrap.cc:492] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:53.300529 3231 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:53.309576 3145 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:53.320287 3231 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.309609 3237 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:53.334782 2642 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:53.335523 2642 ts_tablet_manager.cc:1397] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.435s user 0.028s sys 0.050s
I20260806 03:50:53.300729 3162 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.318414 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:53.317761 3210 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.368458 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:53.376101 2642 raft_consensus.cc:348] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.377490 2642 raft_consensus.cc:374] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:53.377925 2642 raft_consensus.cc:729] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:53.337245 3237 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.396925 2818 consensus_queue.cc:1035] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
W20260806 03:50:53.379900 2332 leader_election.cc:341] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:53.438158 2642 consensus_queue.cc:260] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.363155 3231 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:53.477646 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 97779us
I20260806 03:50:53.501374 2643 tablet_bootstrap.cc:654] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:53.488541 2373 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
I20260806 03:50:53.506055 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:53.504420 3162 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:53.508828 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:53.523605 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:53.528283 2332 leader_election.cc:341] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:53.551391 3210 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:53.554564 3237 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:53.554579 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:53.556193 2480 raft_consensus.cc:2379] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:50:53.560101 3249 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.058s user 0.001s sys 0.000s
I20260806 03:50:53.565716 3218 consensus_queue.cc:1035] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 03:50:53.606336 2644 tablet_bootstrap.cc:654] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:53.609930 2643 tablet_bootstrap.cc:492] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:53.610519 2643 ts_tablet_manager.cc:1397] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.352s user 0.011s sys 0.006s
I20260806 03:50:53.613860 2643 raft_consensus.cc:348] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.614945 2643 raft_consensus.cc:374] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:53.615286 2643 raft_consensus.cc:729] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:53.617238 2643 consensus_queue.cc:260] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:53.619577 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 181277us
I20260806 03:50:53.601368 3255 consensus_queue.cc:1035] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.018s
W20260806 03:50:53.640579 2331 leader_election.cc:341] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:53.644214 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:53.640734 2643 ts_tablet_manager.cc:1428] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.030s user 0.009s sys 0.002s
W20260806 03:50:53.659114 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 133715us
I20260806 03:50:53.652125 2534 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:53.636626 3151 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:53.685200 2643 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:53.686606 2642 ts_tablet_manager.cc:1428] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.351s user 0.028s sys 0.011s
I20260806 03:50:53.673156 3015 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:53.678167 3145 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:53.703878 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:53.689507 3224 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:53.685914 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:53.717387 2644 tablet_bootstrap.cc:492] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:53.718201 2644 ts_tablet_manager.cc:1397] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.411s user 0.037s sys 0.053s
I20260806 03:50:53.677291 3151 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.712303 3224 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:53.725191 3224 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:53.720175 2373 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:50:53.743723 2643 tablet_bootstrap.cc:654] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:53.738292 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:53.746301 2370 raft_consensus.cc:2379] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 1: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:50:53.748005 3145 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:53.749917 3151 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:53.731865 3218 consensus_queue.cc:1035] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260806 03:50:53.755309 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:53.756870 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:53.757537 2572 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:53.755340 2644 raft_consensus.cc:348] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.758848 2644 raft_consensus.cc:374] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:53.759202 2644 raft_consensus.cc:729] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:53.759955 2644 consensus_queue.cc:260] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.767020 2534 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:53.768369 2818 raft_consensus.cc:3037] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:53.769420 2534 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:53.772653 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:53.777073 2429 consensus_peers.cc:487] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c -> Peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Couldn't send request to peer dff94100ffc5485f92cbe6a85addf0b2. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:53.778029 3219 raft_consensus.cc:2781] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:53.778491 3219 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:53.778851 3219 raft_consensus.cc:3037] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:53.777120 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:53.798969 2538 leader_election.cc:341] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:53.780855 2233 catalog_manager.cc:5495] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:53.801429 2818 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:53.819754 2644 ts_tablet_manager.cc:1428] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.101s user 0.008s sys 0.002s
W20260806 03:50:53.846210 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 72206.9us
W20260806 03:50:53.853366 2534 leader_election.cc:341] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:53.853892 2534 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:50:53.854080 2332 leader_election.cc:341] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:53.854766 3162 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:53.868608 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:53.813591 3148 consensus_queue.cc:1035] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260806 03:50:53.885685 2572 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:53.875101 3089 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:53.897424 3089 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.908703 3089 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:53.905207 3257 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260806 03:50:53.919039 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:53.920145 2481 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:53.923209 2332 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:53.926944 3224 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.928642 2578 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:53.932811 2331 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:53.938360 2642 tablet_bootstrap.cc:492] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:53.938781 2644 tablet_bootstrap.cc:492] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
W20260806 03:50:53.933786 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 69142.9us
I20260806 03:50:53.916499 3219 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.972488 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:53.955940 3233 raft_consensus.cc:2781] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:53.967463 3218 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:53.975499 3218 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:53.973302 2576 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:53.986102 2643 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:53.986862 2643 ts_tablet_manager.cc:1397] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.309s user 0.014s sys 0.026s
I20260806 03:50:53.960319 3268 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:53.979156 3218 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:53.982395 3233 raft_consensus.cc:686] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:53.973902 3224 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:53.991876 3268 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:54.024646 3219 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:54.018692 3268 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:50:54.027983 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 66751.4us
W20260806 03:50:54.038862 3273 log.cc:924] Time spent T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260806 03:50:53.995478 3233 consensus_queue.cc:237] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.105620 2236 catalog_manager.cc:5495] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:54.108662 2236 catalog_manager.cc:5495] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:50:54.142827 2642 tablet_bootstrap.cc:654] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.145692 2643 raft_consensus.cc:348] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.146893 2643 raft_consensus.cc:374] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.147284 2643 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.150517 2481 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:54.148272 2643 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.151072 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:54.165156 2643 ts_tablet_manager.cc:1428] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.178s user 0.010s sys 0.000s
I20260806 03:50:54.169076 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:50:54.172318 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 273611us
I20260806 03:50:54.169538 3162 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260806 03:50:54.177696 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:54.164929 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:54.208890 2479 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:54.226837 2643 tablet_bootstrap.cc:492] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:54.211221 3140 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:54.365687 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 196724us
I20260806 03:50:54.415284 3236 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:54.433348 3236 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.435748 3236 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:54.416004 3235 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:54.436527 3235 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.438632 3235 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:54.416896 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:54.419588 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:54.343861 3229 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:54.444435 3229 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.446739 3229 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:54.447125 2644 tablet_bootstrap.cc:654] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.448771 2479 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:54.451273 2375 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:50:54.459677 2538 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:54.461665 3276 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:54.462206 3276 raft_consensus.cc:686] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:54.463130 3276 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.459712 2643 tablet_bootstrap.cc:654] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.453090 3268 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.492769 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:54.493706 2480 raft_consensus.cc:2379] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:50:54.498124 2436 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:54.505170 2642 tablet_bootstrap.cc:492] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:54.505735 2642 ts_tablet_manager.cc:1397] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.582s user 0.008s sys 0.014s
I20260806 03:50:54.508111 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:54.509194 2583 raft_consensus.cc:2396] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260806 03:50:54.516075 2534 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:54.518399 2644 tablet_bootstrap.cc:492] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:54.519842 2644 ts_tablet_manager.cc:1397] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.581s user 0.013s sys 0.013s
I20260806 03:50:54.529914 3286 raft_consensus.cc:2781] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:54.523759 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:54.535112 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:54.537787 3286 raft_consensus.cc:686] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:54.538829 3286 consensus_queue.cc:237] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.539824 3140 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 03:50:54.541402 3295 log.cc:924] Time spent T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260806 03:50:54.553107 2642 raft_consensus.cc:348] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.572721 2642 raft_consensus.cc:374] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.575043 3268 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:54.573997 2642 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.577212 2642 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.584333 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:54.585093 2581 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:50:54.591135 2332 leader_election.cc:341] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:54.591351 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:54.591930 2332 leader_election.cc:341] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:54.592782 2376 raft_consensus.cc:2396] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 3 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260806 03:50:54.593869 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:54.594571 2478 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:54.603015 2581 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:50:54.604391 2331 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:54.608095 2642 ts_tablet_manager.cc:1428] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.102s user 0.009s sys 0.001s
I20260806 03:50:54.617921 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:54.618657 2577 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:54.619314 2644 raft_consensus.cc:348] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.605382 3233 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260806 03:50:54.626108 2644 raft_consensus.cc:374] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.608588 3231 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:54.628270 2642 tablet_bootstrap.cc:492] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:54.626428 2644 raft_consensus.cc:729] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.634989 2644 consensus_queue.cc:260] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.630858 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:54.635885 2233 catalog_manager.cc:5495] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:54.631510 3231 raft_consensus.cc:686] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:54.625558 2331 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:54.640453 3140 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:54.640877 3140 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:54.641218 3140 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:54.639730 3231 consensus_queue.cc:237] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:54.646979 2332 consensus_peers.cc:487] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:54.648609 2644 ts_tablet_manager.cc:1428] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.128s user 0.008s sys 0.000s
I20260806 03:50:54.652745 2373 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:54.655182 2478 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:50:54.655427 2429 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:50:54.658980 2331 leader_election.cc:341] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:54.659391 2331 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:54.661505 2644 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:54.664388 3235 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:54.668926 3229 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:50:54.739250 2642 tablet_bootstrap.cc:654] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.741231 3235 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:54.741798 3235 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.745433 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:54.760547 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:54.767498 2332 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:54.761569 3278 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:54.737344 3140 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:54.780563 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:54.772962 3236 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:54.761194 3235 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:54.792397 2643 tablet_bootstrap.cc:492] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:54.792971 2643 ts_tablet_manager.cc:1397] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.567s user 0.017s sys 0.018s
I20260806 03:50:54.795873 2644 tablet_bootstrap.cc:654] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.795960 2643 raft_consensus.cc:348] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.796862 2643 raft_consensus.cc:374] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.797142 2643 raft_consensus.cc:729] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.798010 3140 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:54.797919 2643 consensus_queue.cc:260] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.799477 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:54.800709 2643 ts_tablet_manager.cc:1428] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
W20260806 03:50:54.801028 2332 leader_election.cc:341] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:54.799755 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:54.803687 2577 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:54.806991 2643 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:54.823773 2577 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:54.825393 2331 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:54.826606 3235 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:54.827105 3235 raft_consensus.cc:686] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:54.828090 3235 consensus_queue.cc:237] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.833146 2642 tablet_bootstrap.cc:492] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:54.833678 2642 ts_tablet_manager.cc:1397] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.218s user 0.013s sys 0.005s
I20260806 03:50:54.841751 2642 raft_consensus.cc:348] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.843052 2642 raft_consensus.cc:374] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.843319 2642 raft_consensus.cc:729] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.844265 2642 consensus_queue.cc:260] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.846519 2644 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:54.847615 2643 tablet_bootstrap.cc:654] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.848181 2644 ts_tablet_manager.cc:1397] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.188s user 0.014s sys 0.011s
I20260806 03:50:54.850749 2642 ts_tablet_manager.cc:1428] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.017s user 0.007s sys 0.001s
I20260806 03:50:54.851958 2642 tablet_bootstrap.cc:492] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:54.857679 2644 raft_consensus.cc:348] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.858491 2644 raft_consensus.cc:374] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.858781 2644 raft_consensus.cc:729] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.859288 2642 tablet_bootstrap.cc:654] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.859616 2644 consensus_queue.cc:260] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.862463 2644 ts_tablet_manager.cc:1428] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260806 03:50:54.863816 2644 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:54.862466 2643 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:54.865833 2643 ts_tablet_manager.cc:1397] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.059s user 0.011s sys 0.005s
I20260806 03:50:54.868924 2642 tablet_bootstrap.cc:492] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:54.869488 2642 ts_tablet_manager.cc:1397] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.018s user 0.014s sys 0.000s
I20260806 03:50:54.872512 2642 raft_consensus.cc:348] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.873371 2642 raft_consensus.cc:374] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.874433 2642 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.875380 2642 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.875972 2644 tablet_bootstrap.cc:654] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.873394 2643 raft_consensus.cc:348] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.876673 2643 raft_consensus.cc:374] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.877027 2643 raft_consensus.cc:729] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.877785 2643 consensus_queue.cc:260] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.880247 2643 ts_tablet_manager.cc:1428] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.014s user 0.007s sys 0.001s
I20260806 03:50:54.881541 2643 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:54.886795 3276 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:54.884856 2642 ts_tablet_manager.cc:1428] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260806 03:50:54.888496 2642 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:54.896178 2643 tablet_bootstrap.cc:654] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.896467 3276 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:54.907202 2642 tablet_bootstrap.cc:654] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:54.936637 2644 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:54.937208 2644 ts_tablet_manager.cc:1397] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.005s
I20260806 03:50:54.943081 2643 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:54.944422 2643 ts_tablet_manager.cc:1397] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.006s
I20260806 03:50:54.944033 2236 catalog_manager.cc:5495] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:54.944422 2644 raft_consensus.cc:348] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.949316 2644 raft_consensus.cc:374] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.948382 2236 catalog_manager.cc:5495] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 2 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:54.949656 2644 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.950717 2644 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.980255 2644 ts_tablet_manager.cc:1428] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.043s user 0.008s sys 0.001s
I20260806 03:50:54.983914 2643 raft_consensus.cc:348] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.984833 2643 raft_consensus.cc:374] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:54.985077 2643 raft_consensus.cc:729] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:54.986274 2644 tablet_bootstrap.cc:492] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:54.986871 2643 consensus_queue.cc:260] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:54.996877 2642 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:54.997438 2642 ts_tablet_manager.cc:1397] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.009s
I20260806 03:50:54.998406 2643 ts_tablet_manager.cc:1428] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.053s user 0.004s sys 0.004s
I20260806 03:50:55.002470 2643 tablet_bootstrap.cc:492] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:55.009977 3231 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:55.010614 3231 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.011936 2642 raft_consensus.cc:348] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.012697 2642 raft_consensus.cc:374] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:55.012974 2642 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:55.013458 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:55.013622 2642 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.014187 2479 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:55.012713 3231 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:55.027199 2332 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:55.030383 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:55.031167 2583 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:55.038872 2644 tablet_bootstrap.cc:654] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:55.046464 2642 ts_tablet_manager.cc:1428] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.049s user 0.005s sys 0.003s
I20260806 03:50:55.049088 2232 catalog_manager.cc:5495] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:55.050360 2232 catalog_manager.cc:5495] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 2, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 2 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:55.063486 3278 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:55.064163 3278 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.066411 3278 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:55.046664 3235 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:55.069707 3235 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:55.070202 3235 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.070425 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:55.071273 2577 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:55.072286 2374 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.074010 2436 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:55.074869 3278 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:55.075316 3278 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:55.075742 3278 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.077672 2482 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.072328 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:55.080364 2376 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 0.
I20260806 03:50:55.082494 2643 tablet_bootstrap.cc:654] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:55.086352 2642 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:55.089279 2376 raft_consensus.cc:1261] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:55.091040 3270 consensus_queue.cc:1035] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:55.099864 2577 raft_consensus.cc:1261] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 03:50:55.083704 3235 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.101482 3230 consensus_queue.cc:1035] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 03:50:55.106105 3278 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.108359 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:55.113083 3235 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:55.114758 2583 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.115181 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:55.115886 2477 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.118409 2642 tablet_bootstrap.cc:654] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:55.122802 2583 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:55.119727 3278 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:55.125957 2477 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:55.126458 2644 tablet_bootstrap.cc:492] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:55.128175 2644 ts_tablet_manager.cc:1397] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.147s user 0.010s sys 0.007s
I20260806 03:50:55.129034 2331 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:55.130121 3252 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:55.130633 3252 raft_consensus.cc:686] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:55.131461 3252 consensus_queue.cc:237] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.141224 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:55.142179 2583 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.145098 2477 raft_consensus.cc:1261] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:55.152084 3276 consensus_queue.cc:1035] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:55.157401 2583 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:55.167694 2644 raft_consensus.cc:348] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.170387 2644 raft_consensus.cc:374] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:55.170655 2644 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:55.172518 2644 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.185405 2644 ts_tablet_manager.cc:1428] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260806 03:50:55.186344 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:55.187044 2376 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.162147 2436 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:55.202690 2482 raft_consensus.cc:1261] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260806 03:50:55.206408 2644 tablet_bootstrap.cc:492] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2: Bootstrap starting.
I20260806 03:50:55.215058 3231 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:55.243417 3231 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.215190 3227 consensus_queue.cc:1035] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260806 03:50:55.210459 3278 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:55.305212 2642 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:55.310197 3231 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:55.304646 3270 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:55.311126 3270 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.306833 2642 ts_tablet_manager.cc:1397] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.250s user 0.024s sys 0.038s
I20260806 03:50:55.318226 2643 tablet_bootstrap.cc:492] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:55.318838 2643 ts_tablet_manager.cc:1397] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.317s user 0.033s sys 0.026s
W20260806 03:50:55.313657 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 67952.6us
I20260806 03:50:55.335872 2644 tablet_bootstrap.cc:654] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:55.336102 2232 catalog_manager.cc:5495] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:55.343401 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:55.331230 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:55.347853 3227 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:55.348446 3227 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.327399 3089 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20260806 03:50:55.357800 3227 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:55.366261 3270 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:55.372394 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:55.377797 2581 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:55.383929 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:55.386369 2538 leader_election.cc:341] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:55.394021 2331 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:55.399047 2643 raft_consensus.cc:348] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.400723 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:55.401686 2643 raft_consensus.cc:374] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:55.402554 2643 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:55.388808 3278 raft_consensus.cc:686] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:55.401001 2376 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:55.402822 3231 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:55.409044 3231 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:55.409436 3231 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.409722 2642 raft_consensus.cc:348] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.410526 2642 raft_consensus.cc:374] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:55.410810 2642 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:55.411489 2642 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.415622 2572 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:55.423651 2642 ts_tablet_manager.cc:1428] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.110s user 0.007s sys 0.000s
I20260806 03:50:55.404412 2643 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.428429 3278 consensus_queue.cc:237] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.431998 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:55.433959 2643 ts_tablet_manager.cc:1428] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.115s user 0.004s sys 0.004s
I20260806 03:50:55.435313 2643 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:55.436892 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:55.438462 3235 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:55.438860 3235 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:55.439215 3235 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.425261 2642 tablet_bootstrap.cc:492] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Bootstrap starting.
I20260806 03:50:55.443696 2481 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:55.454329 2332 leader_election.cc:341] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:55.465227 2332 leader_election.cc:341] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:55.428817 3231 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.501147 2644 tablet_bootstrap.cc:492] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2: No bootstrap required, opened a new log
I20260806 03:50:55.502904 2644 ts_tablet_manager.cc:1397] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent bootstrapping tablet: real 0.297s user 0.022s sys 0.008s
I20260806 03:50:55.491778 3341 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:55.525938 2368 raft_consensus.cc:2379] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:50:55.522341 3357 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:55.532285 3230 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260806 03:50:55.599835 3231 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:55.536667 3341 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.549255 3357 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:55.662885 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 64563us
I20260806 03:50:55.670358 2642 tablet_bootstrap.cc:654] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Neither blocks nor log segments found. Creating new log.
W20260806 03:50:55.657763 3348 log.cc:924] Time spent T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.118s user 0.000s sys 0.001s
W20260806 03:50:55.716949 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 71942.5us
W20260806 03:50:55.785218 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 100897us
I20260806 03:50:55.794354 3227 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:55.794896 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:55.795601 2577 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:55.811816 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:55.821246 2644 raft_consensus.cc:348] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.853533 2644 raft_consensus.cc:374] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:55.853987 2644 raft_consensus.cc:729] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Initialized, Role: FOLLOWER
I20260806 03:50:55.854858 2644 consensus_queue.cc:260] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:55.886744 2644 ts_tablet_manager.cc:1428] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent starting tablet: real 0.383s user 0.007s sys 0.005s
I20260806 03:50:55.905550 2643 tablet_bootstrap.cc:654] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
W20260806 03:50:56.021942 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260806 03:50:56.044605 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 100752us
W20260806 03:50:55.981927 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 360154us
W20260806 03:50:56.076006 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 109041us
I20260806 03:50:56.095160 2577 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:56.012249 3357 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:55.943176 3341 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.130975 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:56.157392 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 55580.8us
W20260806 03:50:56.173102 3235 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.733s user 0.009s sys 0.007s
I20260806 03:50:56.174502 3089 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:56.175060 3089 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.177232 3089 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.174110 3235 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:56.182734 2332 leader_election.cc:341] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
W20260806 03:50:56.183828 2538 leader_election.cc:341] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:56.192786 2642 tablet_bootstrap.cc:492] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: No bootstrap required, opened a new log
I20260806 03:50:56.193537 2642 ts_tablet_manager.cc:1397] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Time spent bootstrapping tablet: real 0.769s user 0.019s sys 0.016s
I20260806 03:50:56.194460 3377 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260806 03:50:56.189482 3235 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.227162 3358 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:56.228664 3358 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.230366 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:56.233286 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:56.233991 2583 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:56.413434 3368 consensus_queue.cc:1035] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:56.417181 2643 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:56.417961 2643 ts_tablet_manager.cc:1397] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.983s user 0.041s sys 0.005s
I20260806 03:50:56.426223 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:50:56.434270 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 246046us
I20260806 03:50:56.422174 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 356) took 1022 ms. Trace:
W20260806 03:50:56.488256 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 55051.6us
I20260806 03:50:56.489646 3404 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:50:56.489675 2332 leader_election.cc:341] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:56.490760 3151 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:56.488368 3385 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:50:56.492132 3385 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.492745 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:56.490805 3404 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.489686 2642 raft_consensus.cc:348] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.497615 2642 raft_consensus.cc:374] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:56.497922 2642 raft_consensus.cc:729] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Initialized, Role: FOLLOWER
I20260806 03:50:56.498750 2642 consensus_queue.cc:260] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.499179 3404 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.491343 3151 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.500581 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:56.501343 3151 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:56.494285 3385 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:56.503933 3368 consensus_queue.cc:1035] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:56.511099 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:56.513669 2331 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:56.553692 2642 ts_tablet_manager.cc:1428] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Time spent starting tablet: real 0.360s user 0.007s sys 0.008s
I20260806 03:50:56.553941 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:56.557739 3406 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:56.519976 3227 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:56.562573 3406 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:50:56.575685 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 84043.6us
I20260806 03:50:56.569957 3406 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:56.596832 3358 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.595820 2643 raft_consensus.cc:348] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:56.603426 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 48000.9us
I20260806 03:50:56.603609 2643 raft_consensus.cc:374] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:56.604773 2643 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:56.480455 2368 rpcz_store.cc:276] 0806 03:50:55.390342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:55.390819 (+ 477us) service_pool.cc:227] Handling call
0806 03:50:56.397645 (+1006826us) spinlock_profiling.cc:243] Waited 79.2 ms on lock 0x7b54003ea8d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:50:56.411754 (+ 14109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":182220160}
I20260806 03:50:56.607015 2643 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.589905 3383 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:56.605990 3252 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:50:56.621021 3394 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:56.621640 3394 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.622568 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:56.623657 2643 ts_tablet_manager.cc:1428] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.205s user 0.006s sys 0.003s
I20260806 03:50:56.619876 3383 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.636852 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:56.638271 3383 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.629555 3252 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.642786 3252 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.702229 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:56.703109 2643 tablet_bootstrap.cc:492] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:56.703235 2583 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
W20260806 03:50:56.746371 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 126661us
W20260806 03:50:56.754277 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 110715us
I20260806 03:50:56.766983 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:56.630695 3227 raft_consensus.cc:686] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:56.781719 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:56.782687 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:56.782687 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:56.702371 3257 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:56.788287 3422 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:56.803035 3257 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.806509 2429 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:56.804006 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:56.816385 3383 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:56.804199 3422 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.818826 3394 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.819481 3257 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:56.821563 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:56.816418 3227 consensus_queue.cc:237] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.830556 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:56.831825 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:56.833582 2643 tablet_bootstrap.cc:654] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
W20260806 03:50:56.836258 2538 leader_election.cc:341] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:56.829175 3422 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:50:56.837679 2332 leader_election.cc:341] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Illegal state: must be running to vote when last-logged opid is not known
I20260806 03:50:56.847693 3383 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:56.854853 3383 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.858021 3383 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.840548 3406 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.862082 3406 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.911702 3383 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:56.912494 3383 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.915171 3383 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.918017 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:56.938081 3406 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:56.938778 3406 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.941977 3406 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.942829 3406 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:56.943589 3406 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:56.946309 3406 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:56.950623 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:56.951347 2579 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:56.951704 3429 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:50:56.958096 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 105330us
I20260806 03:50:56.966727 2583 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:57.097913 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:56.952477 3429 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:57.103861 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 144515us
W20260806 03:50:57.115238 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 165946us
I20260806 03:50:57.129983 2332 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.133224 2332 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.136440 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.153141 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.154072 2376 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:50:57.154274 3378 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.154507 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 370) took 1044 ms. Trace:
I20260806 03:50:57.154918 2376 rpcz_store.cc:276] 0806 03:50:56.109764 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.109983 (+ 219us) service_pool.cc:227] Handling call
0806 03:50:57.152932 (+1042949us) spinlock_profiling.cc:243] Waited 1.37 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:50:57.154443 (+ 1511us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3147520}
I20260806 03:50:57.136148 3235 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.164080 3405 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.165059 3405 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.167620 2643 tablet_bootstrap.cc:492] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:57.171818 2643 ts_tablet_manager.cc:1397] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.473s user 0.023s sys 0.024s
I20260806 03:50:57.169577 2332 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.176424 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.156191 3283 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.193571 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:57.178088 3231 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.168272 2579 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:57.201752 3417 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:57.201879 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:57.204368 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.207371 2482 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:57.204968 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.208437 2583 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:50:57.224763 2329 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.018s sys 0.042s
W20260806 03:50:57.225339 2329 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.660s user 0.019s sys 0.043s
I20260806 03:50:57.224664 3258 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.235041 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.221141 3140 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.224553 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.264771 2482 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:57.203095 3283 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.270673 3283 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.271653 3343 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.233201 3417 raft_consensus.cc:686] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:57.233655 3405 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.272825 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:57.279840 3439 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260806 03:50:57.280380 3429 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:57.280460 2375 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 0.
W20260806 03:50:57.283878 3431 log.cc:924] Time spent T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
I20260806 03:50:57.264806 3140 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.289608 3419 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.289805 3140 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.290371 3419 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.300791 3440 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.307705 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:57.294962 3283 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.301597 3440 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.259718 3258 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.272271 3343 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.345749 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.412324 2377 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:57.403316 2643 raft_consensus.cc:348] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.419656 2643 raft_consensus.cc:374] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:57.420096 2643 raft_consensus.cc:729] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:57.362037 3407 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:57.402560 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 374) took 1270 ms. Trace:
W20260806 03:50:57.306377 2534 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.761s (SENT) [suppressed 4 similar messages]
I20260806 03:50:57.424672 2643 consensus_queue.cc:260] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.440766 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.416844 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.426324 2374 rpcz_store.cc:276] 0806 03:50:56.131808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.134114 (+ 2306us) service_pool.cc:227] Handling call
0806 03:50:57.401682 (+1267568us) spinlock_profiling.cc:243] Waited 6.2 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:50:57.401957 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:50:57.402170 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:50:57.402199 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:50:57.402239 (+ 40us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:50:57.402273 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:50:57.402339 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:50:57.402459 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14259456}
I20260806 03:50:57.373102 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.412371 3426 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.401420 3419 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.427364 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.453032 2643 ts_tablet_manager.cc:1428] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.280s user 0.008s sys 0.004s
I20260806 03:50:57.417573 3283 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.460539 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.450202 3407 raft_consensus.cc:686] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:57.461481 2574 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 0.
I20260806 03:50:57.463521 2643 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:57.456377 3341 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.464902 3444 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.465111 3341 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.452327 2575 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:57.469391 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.472880 3258 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.481554 2331 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:57.478150 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.480556 3343 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.478993 3283 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.447842 3426 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.498631 3258 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:57.499066 3258 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:57.465538 3444 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.376302 3417 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.486294 3341 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.472133 3407 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.482612 2429 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.497083 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.499430 3258 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:57.502143 3426 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.506691 3444 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:57.507623 2429 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.507725 3341 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:57.507822 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.508334 3341 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.510129 3341 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:57.524693 2332 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.527136 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.511276 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 452) took 1380 ms. Trace:
I20260806 03:50:57.542186 3408 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.522441 3383 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.546397 2480 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:50:57.527467 2332 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.543788 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 470) took 1204 ms. Trace:
I20260806 03:50:57.549701 3218 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:57.567260 3356 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.567718 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:57.568936 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 262591us
I20260806 03:50:57.567420 3218 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.572548 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.576525 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.567777 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.545475 2581 rpcz_store.cc:276] 0806 03:50:56.124326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.124447 (+ 121us) service_pool.cc:227] Handling call
0806 03:50:56.124807 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:50:56.157004 (+ 32197us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:50:56.157034 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:50:57.160864 (+1003830us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b2c001765a8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:50:57.161272 (+ 408us) log.cc:842] Serialized 26 byte log entry
0806 03:50:57.447991 (+286719us) raft_consensus.cc:1631] Marking committed up to 0
0806 03:50:57.448052 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:50:57.448123 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:50:57.500056 (+ 51933us) raft_consensus.cc:1671] finished
0806 03:50:57.500086 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:50:57.505095 (+ 5009us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31723,"raft.queue_time_us":32280,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":123,"spinlock_wait_cycles":10592512,"thread_start_us":286417,"threads_started":1,"wal-append.queue_time_us":310887}
I20260806 03:50:57.588189 2534 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:57.588423 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.588616 3447 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.589082 3429 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:57.589246 3447 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.589924 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.591094 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.591943 2534 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:57.592089 3447 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:57.592902 3429 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.574477 3440 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:57.595319 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.595498 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 462) took 1363 ms. Trace:
I20260806 03:50:57.596975 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.597108 2584 rpcz_store.cc:276] 0806 03:50:56.231660 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.410895 (+179235us) service_pool.cc:227] Handling call
0806 03:50:57.586946 (+1176051us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b5800264c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:50:57.587170 (+ 224us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:50:57.595174 (+ 8004us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:50:57.595202 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:50:57.595248 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:50:57.595279 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:50:57.595333 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:50:57.595437 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":43090688}
I20260806 03:50:57.586218 3343 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.602214 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.576911 2372 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:57.607844 2534 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:50:57.609333 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.609633 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.612294 2429 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:57.613767 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.584440 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.615346 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.618199 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.586969 3218 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.624861 3437 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:57.625334 3437 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.625706 3437 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:57.599012 3231 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.604168 3419 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.632714 2436 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.610623 3383 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.623826 3444 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 03:50:57.633782 3231 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.645936 3444 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:50:57.646459 3444 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:57.643024 3419 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.644966 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.647773 2331 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.643771 3218 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.663084 3419 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.663703 3419 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:57.658739 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:57.623075 3343 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.687184 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.655776 3383 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.549602 2477 rpcz_store.cc:276] 0806 03:50:56.339522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.356513 (+ 16991us) service_pool.cc:227] Handling call
0806 03:50:56.405819 (+ 49306us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:50:56.437876 (+ 32057us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:50:56.444288 (+ 6412us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:50:56.525371 (+ 81083us) log.cc:842] Serialized 26 byte log entry
0806 03:50:56.591615 (+ 66244us) raft_consensus.cc:1631] Marking committed up to 0
0806 03:50:56.593354 (+ 1739us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:50:56.605284 (+ 11930us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:50:57.498869 (+893585us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x7b24001374f8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d799f5cf4 00007f0d799f3dfb 00007f0d79a82559 00007f0d79a88495 00007f0d79a88362 00007f0d79a883ee 00007f0d794c65db 00007f0d819ebdda 00007f0d79a882fc 00007f0d79a7a9bd
0806 03:50:57.543362 (+ 44493us) raft_consensus.cc:1671] finished
0806 03:50:57.543390 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:50:57.543684 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":35454,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":32097,"spinlock_wait_cycles":17263872,"thread_start_us":62113,"threads_started":1,"wal-append.queue_time_us":177265}
I20260806 03:50:57.681420 3231 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.701732 2643 tablet_bootstrap.cc:654] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:57.689497 3385 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.691396 3419 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.726302 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.733894 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:57.708638 3278 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.743086 2236 catalog_manager.cc:5495] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:57.697321 3455 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:50:57.747637 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:57.727788 3227 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.767396 3445 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:57.768124 3445 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.711881 3385 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.725291 3419 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.768528 3445 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:57.784227 3446 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260806 03:50:57.778640 3391 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.790874 3358 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.814735 3358 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.788288 3391 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:57.821648 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 192987us
W20260806 03:50:57.859411 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 239661us
I20260806 03:50:57.767378 3455 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.889134 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 145) took 1246 ms. Trace:
I20260806 03:50:57.760303 3278 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.889684 2575 rpcz_store.cc:276] 0806 03:50:56.642369 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.643924 (+ 1555us) service_pool.cc:227] Handling call
0806 03:50:57.416635 (+772711us) spinlock_profiling.cc:243] Waited 52.9 ms on lock 0x7b5800264c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:50:57.888891 (+472256us) spinlock_profiling.cc:243] Waited 6.67 ms on lock 0x7b54004452d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:50:57.889031 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136971776}
I20260806 03:50:57.856778 3270 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.785442 3227 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.911972 3373 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:57.912909 3373 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.913434 3358 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.908720 3270 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.953231 3278 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.961170 3227 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.961814 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.974624 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 370) took 1552 ms. Trace:
I20260806 03:50:57.963688 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.982539 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:57.951704 3419 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:57.993645 3218 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:57.994385 3341 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:57.970923 3343 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.017648 3218 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.008606 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.000797 3437 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.029501 3391 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:58.031257 3385 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:58.033104 3444 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.034770 2576 raft_consensus.cc:2379] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:58.036242 3444 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:58.015093 3383 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.051007 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.040938 3270 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:58.073750 3218 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.076562 3341 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.079234 3437 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.016280 3373 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:50:57.982227 3455 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:58.090756 3258 consensus_meta.cc:287] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.584s user 0.007s sys 0.012s
I20260806 03:50:58.092190 3258 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.084661 3445 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.099786 3445 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.145781 2372 raft_consensus.cc:1226] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:58.146451 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.151346 2379 raft_consensus.cc:2379] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:50:58.158012 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:57.975241 2476 rpcz_store.cc:276] 0806 03:50:56.422327 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.422533 (+ 206us) service_pool.cc:227] Handling call
0806 03:50:57.109934 (+687401us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:50:57.110217 (+ 283us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:50:57.286671 (+176454us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:50:57.288583 (+ 1912us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:50:57.293357 (+ 4774us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:50:57.295214 (+ 1857us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:50:57.974302 (+679088us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54001dd3d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:50:57.974368 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:50:57.974544 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":377595648}
I20260806 03:50:58.173928 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 381) took 1412 ms. Trace:
I20260806 03:50:58.174281 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.174757 2474 rpcz_store.cc:276] 0806 03:50:56.761032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:57.860968 (+1099936us) service_pool.cc:227] Handling call
0806 03:50:57.906574 (+ 45606us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:50:57.919957 (+ 13383us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:50:57.919999 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:50:57.920079 (+ 80us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:50:57.922003 (+ 1924us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:50:58.173064 (+251061us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b54001ab654. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:50:58.173154 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:50:58.173544 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20459776}
I20260806 03:50:58.045037 2480 raft_consensus.cc:1226] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:58.183859 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.183889 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.183943 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.186654 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:58.187690 2331 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.200107 2483 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:58.209937 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:58.223456 2483 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:50:58.228839 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:58.229300 2538 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:50:58.230392 2581 raft_consensus.cc:2379] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:58.237895 3252 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.242434 2483 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:58.303068 3151 consensus_queue.cc:1046] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 03:50:58.096267 3258 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.308671 3270 consensus_queue.cc:1046] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:58.315119 3257 raft_consensus.cc:3032] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:58.315201 3270 raft_consensus.cc:3032] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:58.315765 3257 raft_consensus.cc:729] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:58.326735 3257 consensus_queue.cc:260] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.337229 3257 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:58.332089 3151 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 03:50:58.339771 3151 raft_consensus.cc:686] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:58.315904 3270 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:58.342136 3151 consensus_queue.cc:237] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.343062 3270 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:58.370323 3413 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.051s sys 0.072s
W20260806 03:50:58.371028 3413 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.694s user 0.051s sys 0.072s
I20260806 03:50:58.373116 3445 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
W20260806 03:50:58.376930 2236 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s user 0.006s sys 0.009s
W20260806 03:50:58.377385 2236 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.622s user 0.006s sys 0.009s
I20260806 03:50:58.389047 2643 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
W20260806 03:50:58.389338 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.501s user 0.036s sys 0.042s
W20260806 03:50:58.377975 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.728s user 0.029s sys 0.037s
W20260806 03:50:58.389684 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.001s sys 0.028s
I20260806 03:50:58.402544 2643 ts_tablet_manager.cc:1397] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.939s user 0.023s sys 0.012s
W20260806 03:50:58.407148 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.771s user 0.029s sys 0.038s
W20260806 03:50:58.407799 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.001s sys 0.029s
W20260806 03:50:58.409276 2377 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.063s sys 0.095s
W20260806 03:50:58.409686 2377 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.679s user 0.063s sys 0.095s
I20260806 03:50:58.400911 3270 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:50:58.401271 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 530302us
W20260806 03:50:58.423143 2583 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.608s user 0.001s sys 0.014s
W20260806 03:50:58.428610 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 749179us
I20260806 03:50:58.421190 2534 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.431453 2332 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:50:58.441699 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 809052us
I20260806 03:50:58.448539 3270 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.449898 2534 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.453231 3270 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.456153 3356 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.466408 2436 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.469231 3462 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:58.469843 3462 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:58.470412 3462 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:50:58.476567 3477 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260806 03:50:58.507507 2534 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.504864 2643 raft_consensus.cc:348] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.511173 2643 raft_consensus.cc:374] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:58.513161 2643 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:58.514986 3375 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.522205 2429 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.520190 2643 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:58.523535 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 57397.8us
I20260806 03:50:58.524462 3455 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:58.529767 2436 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.538319 3455 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.549991 2436 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.554797 3455 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:58.561966 2643 ts_tablet_manager.cc:1428] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.159s user 0.009s sys 0.004s
I20260806 03:50:58.563746 2643 tablet_bootstrap.cc:492] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Bootstrap starting.
I20260806 03:50:58.564400 3462 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.568099 3462 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.578595 2534 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.591184 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.597152 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.597486 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.598410 2372 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:58.598938 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.599327 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:58.600145 2379 raft_consensus.cc:2379] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 2: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:50:58.601423 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:50:58.602378 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 45376.6us
I20260806 03:50:58.602774 3257 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.603309 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.603554 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.604332 2368 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:50:58.605947 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:58.606722 2370 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:58.607957 3462 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:58.608775 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 146) took 1809 ms. Trace:
I20260806 03:50:58.608987 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.604368 3257 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:50:58.611793 3458 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:58.621270 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.622803 2577 rpcz_store.cc:276] 0806 03:50:56.798921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.800697 (+ 1776us) service_pool.cc:227] Handling call
0806 03:50:57.999833 (+1199136us) spinlock_profiling.cc:243] Waited 4.29 ms on lock 0x7b5800264c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:50:58.000139 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:50:58.000437 (+ 298us) raft_consensus.cc:1520] Early marking committed up to index 1
0806 03:50:58.410932 (+410495us) log.cc:842] Serialized 14 byte log entry
0806 03:50:58.578734 (+167802us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:50:58.608230 (+ 29496us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:50:58.608327 (+ 97us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:50:58.608405 (+ 78us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:50:58.608560 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":351,"prepare.queue_time_us":369465,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":3983,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"spinlock_wait_cycles":9867520,"thread_start_us":510878,"threads_started":2,"wal-append.queue_time_us":195699}
I20260806 03:50:58.620081 3462 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.622783 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.629616 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.630563 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.629626 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.633908 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:58.644670 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:58.645068 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.645797 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.644798 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.659566 2584 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:50:58.661070 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.667785 2429 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:58.668987 3455 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:58.669519 3455 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:58.669981 3455 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:58.674913 2643 tablet_bootstrap.cc:654] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Neither blocks nor log segments found. Creating new log.
I20260806 03:50:58.677963 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.671075 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.681073 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:58.681937 2377 raft_consensus.cc:2379] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 1: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:50:58.683836 3396 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:58.684609 2370 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:50:58.684535 3396 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.686794 3396 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.687764 2578 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:50:58.687826 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.690356 2332 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.691270 3445 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.692404 2377 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:58.700788 3462 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.704174 3396 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:58.708654 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.709252 3283 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:58.722847 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:58.722572 3283 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.711721 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.723835 2477 raft_consensus.cc:2379] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:50:58.721068 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:58.723071 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:50:58.710520 2534 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.733242 2478 raft_consensus.cc:2379] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 1: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:50:58.757069 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:50:58.758304 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 48119.2us
I20260806 03:50:58.759486 2534 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.760636 3257 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 03:50:58.762010 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.765497 2429 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:58.770002 3283 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.770826 3283 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:50:58.771171 3283 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:58.744913 3396 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.773267 2479 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:58.737174 3270 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.777611 3396 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.765743 2534 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.779276 3270 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.767604 2429 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:50:58.785753 3458 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:58.786397 3462 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:58.786507 3458 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.788168 3462 raft_consensus.cc:686] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:58.789806 3458 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.789405 3462 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.791844 3458 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:50:58.771963 3283 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:58.792546 3458 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.805369 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.806143 2534 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.807785 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:58.809038 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.813444 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.804435 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.818054 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.821506 2534 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:50:58.823484 2332 consensus_peers.cc:487] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 -> Peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Couldn't send request to peer 608e57d72aae496aaf8613622153aa0c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:50:58.823897 2643 tablet_bootstrap.cc:492] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: No bootstrap required, opened a new log
I20260806 03:50:58.824429 2643 ts_tablet_manager.cc:1397] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Time spent bootstrapping tablet: real 0.261s user 0.013s sys 0.015s
I20260806 03:50:58.827481 2643 raft_consensus.cc:348] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.828289 2643 raft_consensus.cc:374] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 03:50:58.828583 2643 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Initialized, Role: FOLLOWER
I20260806 03:50:58.829264 2643 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.831486 3456 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.843408 2534 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:58.854471 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.860467 3463 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:50:58.878059 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 65604.6us
I20260806 03:50:58.849220 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:58.883445 3455 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:50:58.888803 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 53031.5us
I20260806 03:50:58.887048 2436 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.892432 2332 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:50:58.897396 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 63329.2us
I20260806 03:50:58.901649 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:58.888104 3458 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.916201 2480 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:58.921778 2436 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.925639 3459 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:50:58.925896 3396 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.926620 3419 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.934809 3257 consensus_queue.cc:1046] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:58.936126 3457 raft_consensus.cc:3032] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:58.936501 3457 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:58.927251 3444 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.969856 2332 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.981220 3404 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:58.937544 3457 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.968281 3375 consensus_queue.cc:1035] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260806 03:50:58.987885 2332 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:58.997704 3341 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:50:58.998765 3455 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:58.998683 3341 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:58.996399 3356 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.014529 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:59.024973 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.029255 3356 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.033079 2332 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.018725 3493 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.044252 3459 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 03:50:59.048501 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:50:59.044740 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 73536.5us
I20260806 03:50:59.052783 3459 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:59.053584 3445 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:50:59.058961 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 85721.5us
I20260806 03:50:59.059854 2436 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.072716 3498 consensus_queue.cc:1035] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:59.076906 2643 ts_tablet_manager.cc:1428] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Time spent starting tablet: real 0.252s user 0.024s sys 0.012s
I20260806 03:50:59.080799 2236 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33778 (request call id 102) took 3131 ms. Trace:
I20260806 03:50:59.083837 3341 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:59.086012 3462 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:59.086443 3496 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:59.081933 3493 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.087527 3496 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.089861 3496 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:59.087477 3462 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.094548 2436 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:59.095280 3496 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:59.095796 2436 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.081293 2236 rpcz_store.cc:276] 0806 03:50:56.008169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:50:56.066717 (+ 58548us) service_pool.cc:227] Handling call
0806 03:50:59.080622 (+3013905us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:50:58.378903 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:50:58.380778 (+ 1875us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:50:58.380839 (+ 61us) write_op.cc:435] Acquired schema lock
0806 03:50:58.380878 (+ 39us) tablet.cc:660] Decoding operations
0806 03:50:58.381450 (+ 572us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:50:58.381558 (+ 108us) write_op.cc:641] Partition lock acquired for write op
0806 03:50:58.381603 (+ 45us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:50:58.381925 (+ 322us) tablet.cc:699] Row locks acquired
0806 03:50:58.381953 (+ 28us) write_op.cc:260] PREPARE: finished
0806 03:50:58.382154 (+ 201us) write_op.cc:270] Start()
0806 03:50:58.382311 (+ 157us) write_op.cc:276] Timestamp: P: 1785988258382091 usec, L: 0
0806 03:50:58.382342 (+ 31us) op_driver.cc:331] REPLICATION: starting
0806 03:50:58.383573 (+ 1231us) log.cc:842] Serialized 534 byte log entry
0806 03:50:58.639940 (+256367us) op_driver.cc:444] REPLICATION: finished
0806 03:50:58.949448 (+309508us) write_op.cc:301] APPLY: starting
0806 03:50:58.963556 (+ 14108us) tablet.cc:1365] starting BulkCheckPresence
0806 03:50:58.972157 (+ 8601us) tablet.cc:1368] finished BulkCheckPresence
0806 03:50:58.991505 (+ 19348us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:50:58.997591 (+ 6086us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:50:58.998008 (+ 417us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:50:58.998199 (+ 191us) write_op.cc:312] APPLY: finished
0806 03:50:58.998584 (+ 385us) log.cc:842] Serialized 22 byte log entry
0806 03:50:58.998813 (+ 229us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:50:58.998902 (+ 89us) write_op.cc:454] Released schema lock
0806 03:50:58.998968 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0806 03:50:59.000672 (+ 1704us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:50:59.000703 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":302201,"apply.run_cpu_time_us":3729,"apply.run_wall_time_us":58431,"num_ops":1,"prepare.queue_time_us":622968,"prepare.run_cpu_time_us":11959,"prepare.run_wall_time_us":20717,"raft.queue_time_us":161269,"raft.run_cpu_time_us":130623,"raft.run_wall_time_us":157195,"replication_time_us":257449,"thread_start_us":947000,"threads_started":4,"wal-append.queue_time_us":15807}]]}
I20260806 03:50:59.098711 3457 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:59.097431 3496 raft_consensus.cc:686] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:59.099359 3283 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.115643 3493 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.117446 3283 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:59.119200 3462 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:59.115029 3496 consensus_queue.cc:237] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.108412 2436 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.147590 3493 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:50:59.148511 3493 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.151571 3493 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:50:59.159766 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 51444.4us
I20260806 03:50:59.160782 2436 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.162375 3283 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.163082 3283 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.175582 2436 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.180478 2436 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.183560 3496 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.189355 3283 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.197496 3458 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:50:59.198063 3458 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.201365 3458 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:50:59.208859 2478 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:59.209316 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:59.208972 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:59.177107 2232 catalog_manager.cc:5495] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:59.214881 2482 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 0 FOLLOWER]: Advancing to term 1
I20260806 03:50:59.216300 2232 catalog_manager.cc:5495] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:59.219569 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:59.227137 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:59.220906 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:59.228065 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:59.228675 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:59.229132 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:59.229463 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:59.227213 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:50:59.233194 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:59.234337 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:59.234929 2484 raft_consensus.cc:1226] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:50:59.235355 2232 catalog_manager.cc:5495] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:59.236220 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:50:59.248355 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:50:59.248395 3375 consensus_queue.cc:1046] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 03:50:59.251288 3375 raft_consensus.cc:3032] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:59.251652 3375 raft_consensus.cc:729] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:50:59.262212 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:50:59.252648 3375 consensus_queue.cc:260] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.268443 3375 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:59.273115 2429 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.274461 3493 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.282049 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:50:59.284699 2429 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.286110 3493 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.287214 2429 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.289398 3493 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.292577 2429 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.293715 3493 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.273339 2576 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:50:59.300856 2332 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.301795 3445 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260806 03:50:59.305555 2429 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.308106 3341 raft_consensus.cc:2726] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.311066 3405 consensus_queue.cc:1046] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:59.312393 3405 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.313885 3446 raft_consensus.cc:3032] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:50:59.314255 3446 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:50:59.315302 3446 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.318125 3446 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:59.326089 2436 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:50:59.327368 2332 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.328299 3459 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:50:59.329399 3462 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:50:59.329977 3462 raft_consensus.cc:686] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:50:59.330950 3462 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:50:59.331735 2576 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:59.336069 3496 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:59.346006 2332 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:50:59.347411 3145 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:50:59.385262 2302 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260806 03:50:59.384446 2235 catalog_manager.cc:5495] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:50:59.400272 2368 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:59.412528 3462 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:59.423175 2236 catalog_manager.cc:5495] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:59.425803 2236 catalog_manager.cc:5495] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:59.455972 2236 catalog_manager.cc:5495] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:50:59.676435 2369 raft_consensus.cc:1261] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:59.677681 3462 consensus_queue.cc:1035] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:59.698670 2578 raft_consensus.cc:1261] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:59.706429 3500 consensus_queue.cc:1035] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:50:59.852254 2368 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:50:59.859853 2368 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:59.861428 3500 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:50:59.871162 2572 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:50:59.873500 3496 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:00.243512 732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 03:51:00.270536 3526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:51:00.270748 3527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 03:51:00.271685 3529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 03:51:00.273658 732 server_base.cc:1034] running on GCE node
I20260806 03:51:00.275950 732 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260806 03:51:00.276273 732 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260806 03:51:00.276485 732 hybrid_clock.cc:648] HybridClock initialized: now 1785988260276468 us; error 0 us; skew 500 ppm
I20260806 03:51:00.277397 732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 03:51:00.288443 732 webserver.cc:460] Webserver started at http://127.0.183.4:33561/ using document root <none> and password file <none>
I20260806 03:51:00.289104 732 fs_manager.cc:359] Metadata directory not provided
I20260806 03:51:00.289422 732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 03:51:00.289786 732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 03:51:00.291257 732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-3-root/instance:
uuid: "cf6042e85bef47c4aac9d064d91501d1"
format_stamp: "Formatted at 2026-08-06 03:51:00 on dist-test-slave-kvfs"
I20260806 03:51:00.297122 732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 03:51:00.301314 3539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 03:51:00.302459 732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 03:51:00.302959 732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-3-root
uuid: "cf6042e85bef47c4aac9d064d91501d1"
format_stamp: "Formatted at 2026-08-06 03:51:00 on dist-test-slave-kvfs"
I20260806 03:51:00.303370 732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 03:51:00.335515 732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 03:51:00.337265 732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 03:51:00.338882 732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 03:51:00.341991 732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 03:51:00.342236 732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 03:51:00.342505 732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 03:51:00.342694 732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 03:51:00.460390 732 rpc_server.cc:307] RPC server started. Bound to: 127.0.183.4:33215
I20260806 03:51:00.460522 3624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.183.4:33215 every 8 connection(s)
I20260806 03:51:00.486441 732 auto_leader_rebalancer-test.cc:413] add a tserver: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:00.499768 3625 heartbeater.cc:347] Connected to a master server at 127.0.183.62:40035
I20260806 03:51:00.500411 3625 heartbeater.cc:464] Registering TS with master...
I20260806 03:51:00.501421 3625 heartbeater.cc:511] Master 127.0.183.62:40035 requested a full tablet report, sending...
I20260806 03:51:00.503986 2236 ts_manager.cc:194] Registered new tserver with Master: cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:00.506423 2236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:38220
I20260806 03:51:00.541802 3444 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:00.542552 3444 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.544641 3444 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:00.545864 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:00.545768 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:00.546674 2476 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:00.547950 2538 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:00.548884 3444 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:00.549268 3444 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:00.549589 3444 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:00.556576 3444 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.558437 3444 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:00.559290 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:00.560792 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:00.561466 2476 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:00.568568 2476 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:00.569581 2538 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:00.570408 3375 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:00.570878 3375 raft_consensus.cc:686] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:00.571769 3375 consensus_queue.cc:237] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.587158 2236 catalog_manager.cc:5495] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:51:00.666074 3356 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:00.666697 3356 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.668659 3356 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:00.669430 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:00.669600 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:00.670258 2476 raft_consensus.cc:2354] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:00.671355 2332 leader_election.cc:398] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:00.671607 2332 leader_election.cc:401] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:51:00.672231 3356 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:51:00.673312 3375 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:00.673926 3375 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.675869 3375 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:00.676684 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:00.676959 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:00.677647 2368 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:00.677860 2476 raft_consensus.cc:2379] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:00.679651 2534 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:00.682222 3444 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:00.682704 3444 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:00.683169 3444 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:00.687628 3356 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:00.691987 3444 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.695186 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:00.695602 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:00.696506 2476 raft_consensus.cc:2379] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:00.694283 3444 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:00.704460 2368 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:00.705651 2534 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:00.706393 3444 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:00.706879 3444 raft_consensus.cc:686] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:00.707815 3444 consensus_queue.cc:237] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.725227 2236 catalog_manager.cc:5495] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 2 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:51:00.964793 3493 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:00.965670 3493 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.968561 3493 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:00.969317 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:00.970032 2369 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:00.971099 2429 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:00.971905 3496 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:00.972340 3496 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:00.972702 3496 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:00.973162 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:00.981027 3496 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:00.982915 3496 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:00.983835 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:00.983942 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:00.984771 2368 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:00.991731 2368 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:00.992835 2429 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:00.993525 3496 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:00.994053 3496 raft_consensus.cc:686] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:00.994995 3496 consensus_queue.cc:237] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:01.006645 2236 catalog_manager.cc:5495] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 3, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 3 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:01.044375 2481 raft_consensus.cc:1261] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:01.046268 3375 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:01.062318 2369 raft_consensus.cc:1261] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:01.066548 3444 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:01.120978 2369 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:01.122781 3444 consensus_queue.cc:1046] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:01.123715 3444 raft_consensus.cc:3032] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:01.124081 3444 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:01.125025 3444 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:01.126237 3444 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:01.392793 2579 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 03:51:01.393342 2579 pending_rounds.cc:76] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 1
I20260806 03:51:01.393618 2579 pending_rounds.cc:97] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 03:51:01.393968 2579 raft_consensus.cc:2866] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:01.395819 3496 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:01.419474 2368 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:01.422899 3496 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260806 03:51:01.510823 3625 heartbeater.cc:503] Master 127.0.183.62:40035 was elected leader, sending a full tablet report...
I20260806 03:51:01.662182 2368 consensus_queue.cc:237] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:01.675050 2482 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:01.676589 3356 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:01.713289 2579 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:01.721010 3641 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:01.723588 3356 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:01.725546 2482 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
W20260806 03:51:01.753535 2331 consensus_peers.cc:487] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 95f6f3de31d442cfa812406b89ee2fe3. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:01.753847 2372 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:01.754482 2232 catalog_manager.cc:5495] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:01.763168 2578 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
W20260806 03:51:01.779107 2331 consensus_peers.cc:487] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9f040dd271944b9bf08554a90afa940. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:01.781351 2578 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:01.782346 2482 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:01.783051 3356 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:01.784232 3641 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:01.805243 3356 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:01.809505 2482 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:01.809650 2578 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:01.851235 2236 catalog_manager.cc:5495] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:01.864218 2578 consensus_queue.cc:237] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:01.876401 2482 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:01.876475 2372 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:01.877902 3257 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:01.881232 3444 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:01.894157 3375 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:01.910846 2372 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:01.939203 2236 catalog_manager.cc:5495] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:01.940348 2481 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
W20260806 03:51:01.955816 2537 consensus_peers.cc:487] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5489408bdcc042f3919eae139ea82e7a. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:01.956737 2476 consensus_queue.cc:237] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:01.968627 2578 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:01.979086 2372 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:01.984503 3493 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:01.985635 3496 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:02.006047 3493 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:02.008945 2368 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:02.011353 2578 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:02.027148 2235 catalog_manager.cc:5495] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
W20260806 03:51:02.012885 2436 consensus_peers.cc:487] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5efa489aaa94f9988b77e5efc032e5e. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:02.098611 3693 ts_tablet_manager.cc:927] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:02.103720 3693 tablet_copy_client.cc:316] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:02.124564 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet 95f6f3de31d442cfa812406b89ee2fe3 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:02.125367 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet 95f6f3de31d442cfa812406b89ee2fe3 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-95f6f3de31d442cfa812406b89ee2fe3
I20260806 03:51:02.145522 2392 tablet_copy_source_session.cc:205] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:02.162797 3693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95f6f3de31d442cfa812406b89ee2fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:02.185144 3693 tablet_copy_client.cc:791] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:02.185925 3693 tablet_copy_client.cc:655] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:02.191879 3693 tablet_copy_client.cc:531] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:02.201799 3693 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:02.227187 3693 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:02.228274 3693 tablet_bootstrap.cc:492] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:02.229022 3693 ts_tablet_manager.cc:1397] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.027s user 0.019s sys 0.009s
I20260806 03:51:02.231952 3693 raft_consensus.cc:348] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:02.232934 3693 raft_consensus.cc:729] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:02.233589 3693 consensus_queue.cc:260] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:02.242304 3693 ts_tablet_manager.cc:1428] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260806 03:51:02.244432 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-95f6f3de31d442cfa812406b89ee2fe3 received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:02.245065 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-95f6f3de31d442cfa812406b89ee2fe3 on tablet 95f6f3de31d442cfa812406b89ee2fe3 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:02.379628 3693 ts_tablet_manager.cc:927] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:02.381587 3693 tablet_copy_client.cc:316] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:02.399211 3701 ts_tablet_manager.cc:927] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:02.402309 2598 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet 5489408bdcc042f3919eae139ea82e7a from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:02.403323 2598 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet 5489408bdcc042f3919eae139ea82e7a from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-5489408bdcc042f3919eae139ea82e7a
I20260806 03:51:02.420454 3703 ts_tablet_manager.cc:927] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:02.426759 3701 tablet_copy_client.cc:316] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:02.428555 2598 tablet_copy_source_session.cc:205] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:02.428994 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet e9f040dd271944b9bf08554a90afa940 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:02.429587 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet e9f040dd271944b9bf08554a90afa940 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-e9f040dd271944b9bf08554a90afa940
I20260806 03:51:02.432452 3703 tablet_copy_client.cc:316] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.2:45689
I20260806 03:51:02.432698 3693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5489408bdcc042f3919eae139ea82e7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:02.438700 2392 tablet_copy_source_session.cc:205] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:02.451332 3701 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9f040dd271944b9bf08554a90afa940. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:02.462131 2496 tablet_copy_service.cc:140] P 608e57d72aae496aaf8613622153aa0c: Received BeginTabletCopySession request for tablet f5efa489aaa94f9988b77e5efc032e5e from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56244)
I20260806 03:51:02.462939 2496 tablet_copy_service.cc:161] P 608e57d72aae496aaf8613622153aa0c: Beginning new tablet copy session on tablet f5efa489aaa94f9988b77e5efc032e5e from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56244: session id = cf6042e85bef47c4aac9d064d91501d1-f5efa489aaa94f9988b77e5efc032e5e
I20260806 03:51:02.477102 2496 tablet_copy_source_session.cc:205] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:02.480993 3703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5efa489aaa94f9988b77e5efc032e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:02.497457 3703 tablet_copy_client.cc:791] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:02.498104 3703 tablet_copy_client.cc:655] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:02.502055 3703 tablet_copy_client.cc:531] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:02.504485 3693 tablet_copy_client.cc:791] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:02.505460 3693 tablet_copy_client.cc:655] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:02.511412 3693 tablet_copy_client.cc:531] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:02.518569 3701 tablet_copy_client.cc:791] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:02.519279 3701 tablet_copy_client.cc:655] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:02.521184 3703 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:02.521545 3693 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:02.532153 3701 tablet_copy_client.cc:531] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:02.559897 3701 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:02.565966 3703 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:02.566838 3703 tablet_bootstrap.cc:492] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:02.567050 3693 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:02.567610 3703 ts_tablet_manager.cc:1397] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.047s user 0.017s sys 0.006s
I20260806 03:51:02.568065 3693 tablet_bootstrap.cc:492] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:02.568831 3693 ts_tablet_manager.cc:1397] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.048s user 0.022s sys 0.010s
I20260806 03:51:02.571290 3703 raft_consensus.cc:348] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:02.571393 3693 raft_consensus.cc:348] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:02.572278 3703 raft_consensus.cc:729] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:02.572949 3693 raft_consensus.cc:729] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:02.573696 3594 raft_consensus.cc:1203] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:02.573711 3693 consensus_queue.cc:260] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:02.584945 3703 consensus_queue.cc:260] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:02.594755 3703 ts_tablet_manager.cc:1428] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.027s user 0.010s sys 0.001s
I20260806 03:51:02.603680 2496 tablet_copy_service.cc:342] P 608e57d72aae496aaf8613622153aa0c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-f5efa489aaa94f9988b77e5efc032e5e received from {username='slave'} at 127.0.0.1:56244
I20260806 03:51:02.604189 2496 tablet_copy_service.cc:433] P 608e57d72aae496aaf8613622153aa0c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-f5efa489aaa94f9988b77e5efc032e5e on tablet f5efa489aaa94f9988b77e5efc032e5e with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:02.622335 3693 ts_tablet_manager.cc:1428] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.053s user 0.019s sys 0.031s
I20260806 03:51:02.625967 2598 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-5489408bdcc042f3919eae139ea82e7a received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:02.626438 2598 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-5489408bdcc042f3919eae139ea82e7a on tablet 5489408bdcc042f3919eae139ea82e7a with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:02.642480 3701 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:02.643591 3701 tablet_bootstrap.cc:492] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:02.644374 3701 ts_tablet_manager.cc:1397] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.085s user 0.025s sys 0.012s
I20260806 03:51:02.647200 3701 raft_consensus.cc:348] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:02.648133 3701 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:02.648813 3701 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:02.651240 3701 ts_tablet_manager.cc:1428] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 03:51:02.653712 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-e9f040dd271944b9bf08554a90afa940 received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:02.654168 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-e9f040dd271944b9bf08554a90afa940 on tablet e9f040dd271944b9bf08554a90afa940 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:02.779497 3594 raft_consensus.cc:1203] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:02.807024 3594 raft_consensus.cc:1203] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:02.870671 3594 raft_consensus.cc:1203] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:03.008417 3661 raft_consensus.cc:1050] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:03.011620 3661 consensus_queue.cc:237] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:03.022658 3594 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.024521 3641 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:03.025820 2572 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.028846 3458 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:03.048316 2479 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.050448 3641 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:03.073964 3641 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.082369 2479 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.084682 3594 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.097455 2572 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.106601 2236 catalog_manager.cc:5495] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:03.148141 2372 consensus_queue.cc:237] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:03.153661 2479 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:03.153666 3594 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:03.155223 3356 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:03.156242 3641 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:03.170468 3594 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.170857 2479 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.166551 3661 raft_consensus.cc:2932] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.188203 2233 catalog_manager.cc:5495] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.190330 2215 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 95f6f3de31d442cfa812406b89ee2fe3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:03.204502 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet 95f6f3de31d442cfa812406b89ee2fe3 with delete_type TABLET_DATA_TOMBSTONED (TS fe3d17f4ad984b9088a268d58d87469c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44180
I20260806 03:51:03.211838 3731 tablet_replica.cc:330] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: stopping tablet replica
I20260806 03:51:03.214002 3731 raft_consensus.cc:2229] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:03.214926 3731 raft_consensus.cc:2258] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:03.220377 3731 ts_tablet_manager.cc:1905] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:03.241516 3731 ts_tablet_manager.cc:1918] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:03.242245 3731 log.cc:1196] T 95f6f3de31d442cfa812406b89ee2fe3 P fe3d17f4ad984b9088a268d58d87469c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/wals/95f6f3de31d442cfa812406b89ee2fe3
I20260806 03:51:03.244180 2217 catalog_manager.cc:4841] TS fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): tablet 95f6f3de31d442cfa812406b89ee2fe3 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:03.273495 3710 raft_consensus.cc:1050] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:03.276811 3710 consensus_queue.cc:237] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:03.294421 3652 raft_consensus.cc:1050] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:03.297156 3652 consensus_queue.cc:237] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:03.320775 3594 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.322175 2479 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.323254 3594 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.325646 2572 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.326550 2372 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.324587 3686 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:03.328418 3444 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:03.329072 3493 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:03.343493 3713 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260806 03:51:03.362155 3356 raft_consensus.cc:1050] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:03.359753 3652 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:03.365330 2372 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260806 03:51:03.365926 3733 log.cc:924] Time spent T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260806 03:51:03.365410 3356 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:03.370921 3686 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:03.382648 3593 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.385221 3661 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:03.399276 2482 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.402410 2578 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:03.405073 3458 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:03.409686 3718 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:03.409611 3686 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.413878 3592 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.439370 2232 catalog_manager.cc:5495] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.444072 2481 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.465967 3661 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.470148 2480 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.487095 3592 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.496747 2232 catalog_manager.cc:5495] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:03.508905 3652 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.523772 2572 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.556511 2372 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.574250 3592 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.597802 2368 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.605650 2232 catalog_manager.cc:5495] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.601414 2576 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.617075 2377 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:03.622591 3592 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:03.627291 3661 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:03.649547 2578 consensus_queue.cc:237] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:03.656126 3458 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.658007 3592 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.661310 3593 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:03.643033 2481 consensus_queue.cc:237] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:03.662778 2482 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:03.664041 3444 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:03.664829 3739 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:03.674474 3737 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.677405 3593 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:03.679181 3652 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:03.678689 2232 catalog_manager.cc:5495] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.688261 2480 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.692862 3713 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.698598 2472 tablet_service.cc:1514] Processing DeleteTablet for tablet e9f040dd271944b9bf08554a90afa940 with delete_type TABLET_DATA_TOMBSTONED (TS 608e57d72aae496aaf8613622153aa0c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54212
I20260806 03:51:03.713073 3592 raft_consensus.cc:2932] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.723093 2572 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:03.725111 2215 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e9f040dd271944b9bf08554a90afa940 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:03.731159 2217 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5489408bdcc042f3919eae139ea82e7a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:03.731711 3641 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:03.737118 3752 tablet_replica.cc:330] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: stopping tablet replica
I20260806 03:51:03.737646 2377 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:03.755115 2578 raft_consensus.cc:2932] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.760288 3752 raft_consensus.cc:2229] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:03.761332 3752 raft_consensus.cc:2258] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:03.761652 2232 catalog_manager.cc:5495] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.768747 3592 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:03.776599 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 70217.9us
I20260806 03:51:03.783015 2218 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f5efa489aaa94f9988b77e5efc032e5e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:03.784440 3493 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:03.797508 3752 ts_tablet_manager.cc:1905] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:03.803828 2372 raft_consensus.cc:2932] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:03.822220 2359 tablet_service.cc:1514] Processing DeleteTablet for tablet 5489408bdcc042f3919eae139ea82e7a with delete_type TABLET_DATA_TOMBSTONED (TS dff94100ffc5485f92cbe6a85addf0b2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40712
I20260806 03:51:03.832628 2236 catalog_manager.cc:5495] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:03.843053 3752 ts_tablet_manager.cc:1918] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:03.843887 3752 log.cc:1196] T e9f040dd271944b9bf08554a90afa940 P 608e57d72aae496aaf8613622153aa0c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/wals/e9f040dd271944b9bf08554a90afa940
I20260806 03:51:03.844200 3753 tablet_replica.cc:330] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: stopping tablet replica
I20260806 03:51:03.846577 2218 catalog_manager.cc:4841] TS 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): tablet e9f040dd271944b9bf08554a90afa940 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:03.851524 3753 raft_consensus.cc:2229] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:03.852883 3753 raft_consensus.cc:2258] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:03.859190 3753 ts_tablet_manager.cc:1905] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:03.871765 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet f5efa489aaa94f9988b77e5efc032e5e with delete_type TABLET_DATA_TOMBSTONED (TS fe3d17f4ad984b9088a268d58d87469c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44180
I20260806 03:51:03.892707 3754 tablet_replica.cc:330] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: stopping tablet replica
I20260806 03:51:03.894626 3754 raft_consensus.cc:2229] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:03.895689 3754 raft_consensus.cc:2258] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:03.903151 3754 ts_tablet_manager.cc:1905] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:03.904964 3753 ts_tablet_manager.cc:1918] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:03.905653 3753 log.cc:1196] T 5489408bdcc042f3919eae139ea82e7a P dff94100ffc5485f92cbe6a85addf0b2: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root/wals/5489408bdcc042f3919eae139ea82e7a
I20260806 03:51:03.907913 2215 catalog_manager.cc:4841] TS dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): tablet 5489408bdcc042f3919eae139ea82e7a (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:03.925164 3754 ts_tablet_manager.cc:1918] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:03.925864 3754 log.cc:1196] T f5efa489aaa94f9988b77e5efc032e5e P fe3d17f4ad984b9088a268d58d87469c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/wals/f5efa489aaa94f9988b77e5efc032e5e
I20260806 03:51:03.927626 2217 catalog_manager.cc:4841] TS fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): tablet f5efa489aaa94f9988b77e5efc032e5e (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:04.866011 2377 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:04.873387 2481 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:04.873392 2578 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:04.873435 2331 consensus_peers.cc:487] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ead4b99d10d24d13ba7f07dee24f79e3. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:04.874831 3757 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:04.875730 3641 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:04.885260 3757 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:04.894761 2578 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:04.891983 2481 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:04.904469 2480 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:04.916072 2235 catalog_manager.cc:5495] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 03:51:04.960216 2436 consensus_peers.cc:487] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bcedcc1f7e94405bbfa0a2d56aa30fb. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:04.978487 2572 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:51:04.982282 3493 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260806 03:51:05.002832 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 48298.7us
I20260806 03:51:05.006237 2377 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:51:05.007104 3493 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.009364 3713 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:05.010900 2572 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.023866 2372 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.028527 2377 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.055239 2232 catalog_manager.cc:5495] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:05.058781 2482 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:05.059634 2331 consensus_peers.cc:487] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39115846d5144463b3a3a8e09e8b4500. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:05.060711 2572 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:05.060779 3757 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:05.062840 3757 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:05.075174 3757 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.080816 2480 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.091143 2482 consensus_queue.cc:237] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.099540 2235 catalog_manager.cc:5495] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:05.094883 2576 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.143685 2581 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:05.144079 2377 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:05.144205 2436 consensus_peers.cc:487] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ef53ef49663446658709042138ce12f7. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:05.145720 3493 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:05.147918 3766 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:05.164999 3766 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.167258 2377 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.170428 2581 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.192937 2235 catalog_manager.cc:5495] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:05.330485 3785 ts_tablet_manager.cc:927] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:05.332731 3785 tablet_copy_client.cc:316] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:05.334383 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet ead4b99d10d24d13ba7f07dee24f79e3 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:05.334923 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet ead4b99d10d24d13ba7f07dee24f79e3 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-ead4b99d10d24d13ba7f07dee24f79e3
I20260806 03:51:05.345671 2392 tablet_copy_source_session.cc:205] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:05.349206 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ead4b99d10d24d13ba7f07dee24f79e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:05.368319 3785 tablet_copy_client.cc:791] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:05.368992 3785 tablet_copy_client.cc:655] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:05.373005 3785 tablet_copy_client.cc:531] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:05.382488 3785 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:05.428181 3785 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:05.429478 3785 tablet_bootstrap.cc:492] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:05.431721 3785 ts_tablet_manager.cc:1397] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.049s user 0.015s sys 0.029s
I20260806 03:51:05.437008 3785 raft_consensus.cc:348] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.438158 3785 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:05.439106 3785 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.450534 3785 ts_tablet_manager.cc:1428] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260806 03:51:05.453182 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-ead4b99d10d24d13ba7f07dee24f79e3 received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:05.453766 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-ead4b99d10d24d13ba7f07dee24f79e3 on tablet ead4b99d10d24d13ba7f07dee24f79e3 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:05.501580 3785 ts_tablet_manager.cc:927] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:05.504036 3785 tablet_copy_client.cc:316] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.2:45689
I20260806 03:51:05.505664 2496 tablet_copy_service.cc:140] P 608e57d72aae496aaf8613622153aa0c: Received BeginTabletCopySession request for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56244)
I20260806 03:51:05.506304 2496 tablet_copy_service.cc:161] P 608e57d72aae496aaf8613622153aa0c: Beginning new tablet copy session on tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56244: session id = cf6042e85bef47c4aac9d064d91501d1-9bcedcc1f7e94405bbfa0a2d56aa30fb
I20260806 03:51:05.518709 2496 tablet_copy_source_session.cc:205] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:05.522449 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:05.554083 3785 tablet_copy_client.cc:791] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:05.554771 3785 tablet_copy_client.cc:655] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:05.562316 3785 tablet_copy_client.cc:531] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:05.573268 3785 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:05.598817 3785 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:05.599794 3785 tablet_bootstrap.cc:492] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:05.600589 3785 ts_tablet_manager.cc:1397] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.020s
I20260806 03:51:05.603713 3785 raft_consensus.cc:348] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.604730 3785 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:05.605448 3785 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.607990 3785 ts_tablet_manager.cc:1428] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260806 03:51:05.609618 2496 tablet_copy_service.cc:342] P 608e57d72aae496aaf8613622153aa0c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-9bcedcc1f7e94405bbfa0a2d56aa30fb received from {username='slave'} at 127.0.0.1:56244
I20260806 03:51:05.610175 2496 tablet_copy_service.cc:433] P 608e57d72aae496aaf8613622153aa0c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-9bcedcc1f7e94405bbfa0a2d56aa30fb on tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:05.647156 3785 ts_tablet_manager.cc:927] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:05.651237 3785 tablet_copy_client.cc:316] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:05.652851 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet 39115846d5144463b3a3a8e09e8b4500 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:05.653406 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet 39115846d5144463b3a3a8e09e8b4500 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-39115846d5144463b3a3a8e09e8b4500
I20260806 03:51:05.675711 2392 tablet_copy_source_session.cc:205] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:05.679296 3785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39115846d5144463b3a3a8e09e8b4500. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:05.702845 3785 tablet_copy_client.cc:791] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:05.703480 3785 tablet_copy_client.cc:655] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:05.712378 3785 tablet_copy_client.cc:531] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:05.722962 3785 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:05.744650 3785 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:05.746096 3785 tablet_bootstrap.cc:492] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:05.746388 3802 ts_tablet_manager.cc:927] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:05.746922 3785 ts_tablet_manager.cc:1397] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.024s user 0.021s sys 0.003s
I20260806 03:51:05.749897 3785 raft_consensus.cc:348] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.750078 3802 tablet_copy_client.cc:316] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.2:45689
I20260806 03:51:05.750619 3785 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:05.751356 3785 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.752766 2496 tablet_copy_service.cc:140] P 608e57d72aae496aaf8613622153aa0c: Received BeginTabletCopySession request for tablet ef53ef49663446658709042138ce12f7 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56244)
I20260806 03:51:05.753334 2496 tablet_copy_service.cc:161] P 608e57d72aae496aaf8613622153aa0c: Beginning new tablet copy session on tablet ef53ef49663446658709042138ce12f7 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56244: session id = cf6042e85bef47c4aac9d064d91501d1-ef53ef49663446658709042138ce12f7
I20260806 03:51:05.753602 3785 ts_tablet_manager.cc:1428] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260806 03:51:05.756560 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-39115846d5144463b3a3a8e09e8b4500 received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:05.756965 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-39115846d5144463b3a3a8e09e8b4500 on tablet 39115846d5144463b3a3a8e09e8b4500 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:05.765390 2496 tablet_copy_source_session.cc:205] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:05.772109 3802 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef53ef49663446658709042138ce12f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:05.790916 3802 tablet_copy_client.cc:791] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:05.791684 3802 tablet_copy_client.cc:655] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:05.796794 3802 tablet_copy_client.cc:531] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:05.807092 3802 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:05.831347 3802 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:05.832788 3802 tablet_bootstrap.cc:492] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:05.833639 3802 ts_tablet_manager.cc:1397] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.008s
I20260806 03:51:05.836534 3802 raft_consensus.cc:348] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.837584 3802 raft_consensus.cc:729] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:05.838296 3802 consensus_queue.cc:260] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:05.840698 3802 ts_tablet_manager.cc:1428] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260806 03:51:05.845184 2496 tablet_copy_service.cc:342] P 608e57d72aae496aaf8613622153aa0c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-ef53ef49663446658709042138ce12f7 received from {username='slave'} at 127.0.0.1:56244
I20260806 03:51:05.845666 2496 tablet_copy_service.cc:433] P 608e57d72aae496aaf8613622153aa0c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-ef53ef49663446658709042138ce12f7 on tablet ef53ef49663446658709042138ce12f7 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:05.858055 3592 raft_consensus.cc:1203] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:06.065516 3592 raft_consensus.cc:1203] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 03:51:06.171325 3592 raft_consensus.cc:1203] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:06.197768 3592 raft_consensus.cc:1203] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:06.338948 3779 raft_consensus.cc:1050] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:06.341913 3779 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:06.350575 3592 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.352777 2576 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.355541 3806 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:06.357898 3641 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:06.358793 2479 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.360227 3641 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:06.396281 3777 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.397769 2581 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.400245 2482 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.429487 2232 catalog_manager.cc:5495] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.434360 3592 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.474807 2368 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:06.489434 3592 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:06.489434 2581 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:06.492224 3779 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:06.493330 3641 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:06.503196 3806 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.508561 3592 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.511438 2581 raft_consensus.cc:2932] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.525063 2215 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ead4b99d10d24d13ba7f07dee24f79e3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:06.528385 2236 catalog_manager.cc:5495] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.548785 2472 tablet_service.cc:1514] Processing DeleteTablet for tablet ead4b99d10d24d13ba7f07dee24f79e3 with delete_type TABLET_DATA_TOMBSTONED (TS 608e57d72aae496aaf8613622153aa0c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54212
I20260806 03:51:06.551177 3822 tablet_replica.cc:330] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: stopping tablet replica
I20260806 03:51:06.552659 3822 raft_consensus.cc:2229] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:06.553669 3822 raft_consensus.cc:2258] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:06.560040 3822 ts_tablet_manager.cc:1905] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:06.568396 3493 raft_consensus.cc:1050] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:06.570742 3493 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:06.590059 2368 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:51:06.592770 2576 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:51:06.592814 3780 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:06.594365 3822 ts_tablet_manager.cc:1918] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:06.594882 3822 log.cc:1196] T ead4b99d10d24d13ba7f07dee24f79e3 P 608e57d72aae496aaf8613622153aa0c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/wals/ead4b99d10d24d13ba7f07dee24f79e3
I20260806 03:51:06.595293 3766 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:06.597304 2218 catalog_manager.cc:4841] TS 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): tablet ead4b99d10d24d13ba7f07dee24f79e3 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:06.598263 3592 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:51:06.600240 3766 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:06.619366 3766 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.621001 2581 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.633970 3592 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.637737 2369 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.641551 3780 raft_consensus.cc:1050] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:06.643203 2236 catalog_manager.cc:5495] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:06.665060 3780 consensus_queue.cc:237] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:06.670509 2482 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:06.690120 3592 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:06.695983 3493 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:06.703799 2572 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:06.728159 3766 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:06.757342 3766 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.799276 2369 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.801302 3834 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:06.810357 2572 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.812071 3593 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.826505 3820 raft_consensus.cc:1050] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:06.817250 2581 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.831203 3820 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:06.843502 2218 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:06.848335 3593 raft_consensus.cc:2932] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.847033 3836 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:06.864914 2236 catalog_manager.cc:5495] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 4 to 5, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:06.877625 3493 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:06.893303 2359 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb with delete_type TABLET_DATA_TOMBSTONED (TS dff94100ffc5485f92cbe6a85addf0b2 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:40712
I20260806 03:51:06.909135 3594 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.914263 3641 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:06.922334 2480 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.931674 3844 tablet_replica.cc:330] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: stopping tablet replica
I20260806 03:51:06.936538 3820 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260806 03:51:06.952239 3844 raft_consensus.cc:2229] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:06.953277 3844 raft_consensus.cc:2258] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:06.955188 2576 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:06.961124 3493 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.960287 3820 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260806 03:51:06.979082 3592 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.986773 2368 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:06.997897 2236 catalog_manager.cc:5495] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:07.003396 3844 ts_tablet_manager.cc:1905] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:07.013716 3806 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.016520 3593 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.043542 2581 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.065304 2576 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.074231 2235 catalog_manager.cc:5495] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.113991 2482 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.117434 3844 ts_tablet_manager.cc:1918] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 03:51:07.118279 3844 log.cc:1196] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P dff94100ffc5485f92cbe6a85addf0b2: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root/wals/9bcedcc1f7e94405bbfa0a2d56aa30fb
I20260806 03:51:07.105901 2479 consensus_queue.cc:237] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:07.132862 2215 catalog_manager.cc:4841] TS dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): tablet 9bcedcc1f7e94405bbfa0a2d56aa30fb (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:07.141992 3592 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:07.147130 2576 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:07.152074 3836 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:07.153606 3838 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:07.176419 3834 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.185232 2581 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.197144 2218 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ef53ef49663446658709042138ce12f7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:07.194942 3592 raft_consensus.cc:2932] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.218281 2235 catalog_manager.cc:5495] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:07.243675 2359 tablet_service.cc:1514] Processing DeleteTablet for tablet ef53ef49663446658709042138ce12f7 with delete_type TABLET_DATA_TOMBSTONED (TS dff94100ffc5485f92cbe6a85addf0b2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40712
I20260806 03:51:07.245795 3844 tablet_replica.cc:330] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: stopping tablet replica
I20260806 03:51:07.241837 2368 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:07.258668 3844 raft_consensus.cc:2229] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:07.259960 3844 raft_consensus.cc:2258] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:07.264544 3844 ts_tablet_manager.cc:1905] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:07.266304 3592 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:07.268579 3820 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:07.270761 2581 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:07.273439 3806 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:07.292071 3641 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.307657 2215 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 39115846d5144463b3a3a8e09e8b4500 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:07.309206 2581 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.298465 3844 ts_tablet_manager.cc:1918] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:07.316157 3844 log.cc:1196] T ef53ef49663446658709042138ce12f7 P dff94100ffc5485f92cbe6a85addf0b2: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root/wals/ef53ef49663446658709042138ce12f7
I20260806 03:51:07.317108 3592 raft_consensus.cc:2932] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:07.320498 2235 catalog_manager.cc:5495] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:07.326931 2215 catalog_manager.cc:4841] TS dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): tablet ef53ef49663446658709042138ce12f7 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:07.343850 2472 tablet_service.cc:1514] Processing DeleteTablet for tablet 39115846d5144463b3a3a8e09e8b4500 with delete_type TABLET_DATA_TOMBSTONED (TS 608e57d72aae496aaf8613622153aa0c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54212
I20260806 03:51:07.347259 3850 tablet_replica.cc:330] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: stopping tablet replica
I20260806 03:51:07.355325 3850 raft_consensus.cc:2229] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:07.374130 3850 raft_consensus.cc:2258] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:07.386633 3850 ts_tablet_manager.cc:1905] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:07.435859 3850 ts_tablet_manager.cc:1918] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:07.436517 3850 log.cc:1196] T 39115846d5144463b3a3a8e09e8b4500 P 608e57d72aae496aaf8613622153aa0c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/wals/39115846d5144463b3a3a8e09e8b4500
I20260806 03:51:07.438709 2218 catalog_manager.cc:4841] TS 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): tablet 39115846d5144463b3a3a8e09e8b4500 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
W20260806 03:51:08.529022 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 43559.3us
W20260806 03:51:08.529569 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 44882us
I20260806 03:51:08.543399 2479 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:08.605942 2576 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:08.606125 2436 consensus_peers.cc:487] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d93b8d7e5d334d3dac131b9a4c49b103. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:08.608227 2372 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:08.607801 3838 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:08.614993 3834 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:08.631541 3838 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.638789 2576 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.678318 2235 catalog_manager.cc:5495] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.680581 2480 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:08.681901 2377 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.690629 2372 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:51:08.693403 2578 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
W20260806 03:51:08.693565 2436 consensus_peers.cc:487] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 507574ca8b05468888fa8465ccdb2f80. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:08.696404 3869 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:08.698923 3869 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:08.716840 3862 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.725564 2578 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.728878 2377 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.752247 2576 consensus_queue.cc:237] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:08.763422 2537 consensus_peers.cc:487] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2401ce53befd43c6a455e89c5df49a28. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:08.764719 2479 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:08.766373 3861 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:08.776307 2377 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:08.779238 3860 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:08.785034 2236 catalog_manager.cc:5495] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.815644 3860 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.818107 2479 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.824785 2377 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.843612 2578 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:08.849390 2232 catalog_manager.cc:5495] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:08.861505 2481 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:08.864297 2368 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:08.866110 3853 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 03:51:08.867151 2537 consensus_peers.cc:487] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c210749b55e24fa0bb7b9c7cdf3cc6b0. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:08.869908 3860 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:08.890925 3860 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.902491 2481 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.922367 2377 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:08.927318 2232 catalog_manager.cc:5495] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:09.080708 3890 ts_tablet_manager.cc:927] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:09.083545 3890 tablet_copy_client.cc:316] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.2:45689
I20260806 03:51:09.085795 2496 tablet_copy_service.cc:140] P 608e57d72aae496aaf8613622153aa0c: Received BeginTabletCopySession request for tablet d93b8d7e5d334d3dac131b9a4c49b103 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56244)
I20260806 03:51:09.086408 2496 tablet_copy_service.cc:161] P 608e57d72aae496aaf8613622153aa0c: Beginning new tablet copy session on tablet d93b8d7e5d334d3dac131b9a4c49b103 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56244: session id = cf6042e85bef47c4aac9d064d91501d1-d93b8d7e5d334d3dac131b9a4c49b103
I20260806 03:51:09.096285 2496 tablet_copy_source_session.cc:205] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:09.100739 3890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d93b8d7e5d334d3dac131b9a4c49b103. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:09.118494 3890 tablet_copy_client.cc:791] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:09.119402 3890 tablet_copy_client.cc:655] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:09.123735 3890 tablet_copy_client.cc:531] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:09.127269 3892 ts_tablet_manager.cc:927] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:09.130219 3892 tablet_copy_client.cc:316] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.2:45689
I20260806 03:51:09.131943 2496 tablet_copy_service.cc:140] P 608e57d72aae496aaf8613622153aa0c: Received BeginTabletCopySession request for tablet 507574ca8b05468888fa8465ccdb2f80 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56244)
I20260806 03:51:09.132553 2496 tablet_copy_service.cc:161] P 608e57d72aae496aaf8613622153aa0c: Beginning new tablet copy session on tablet 507574ca8b05468888fa8465ccdb2f80 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56244: session id = cf6042e85bef47c4aac9d064d91501d1-507574ca8b05468888fa8465ccdb2f80
I20260806 03:51:09.135699 3890 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:09.144194 2496 tablet_copy_source_session.cc:205] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:09.147907 3892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 507574ca8b05468888fa8465ccdb2f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:09.200502 3895 ts_tablet_manager.cc:927] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:09.216670 3892 tablet_copy_client.cc:791] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:09.217482 3892 tablet_copy_client.cc:655] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:09.223932 3895 tablet_copy_client.cc:316] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:09.225185 3892 tablet_copy_client.cc:531] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:09.233953 2598 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet 2401ce53befd43c6a455e89c5df49a28 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:09.234719 2598 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet 2401ce53befd43c6a455e89c5df49a28 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-2401ce53befd43c6a455e89c5df49a28
I20260806 03:51:09.257827 2598 tablet_copy_source_session.cc:205] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:09.264700 3892 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:09.265952 3895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2401ce53befd43c6a455e89c5df49a28. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:51:09.334797 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 43490.3us
I20260806 03:51:09.344848 3895 tablet_copy_client.cc:791] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:09.345762 3895 tablet_copy_client.cc:655] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:09.346925 3892 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:09.348595 3892 tablet_bootstrap.cc:492] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:09.350754 3892 ts_tablet_manager.cc:1397] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.086s user 0.031s sys 0.036s
I20260806 03:51:09.354058 3892 raft_consensus.cc:348] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:09.355218 3895 tablet_copy_client.cc:531] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:09.355032 3892 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:09.358009 3892 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:09.368521 3892 ts_tablet_manager.cc:1428] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
I20260806 03:51:09.371076 2496 tablet_copy_service.cc:342] P 608e57d72aae496aaf8613622153aa0c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-507574ca8b05468888fa8465ccdb2f80 received from {username='slave'} at 127.0.0.1:56244
I20260806 03:51:09.371590 2496 tablet_copy_service.cc:433] P 608e57d72aae496aaf8613622153aa0c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-507574ca8b05468888fa8465ccdb2f80 on tablet 507574ca8b05468888fa8465ccdb2f80 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:09.378978 3895 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:09.396390 3890 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:09.397174 3892 ts_tablet_manager.cc:927] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:09.397456 3890 tablet_bootstrap.cc:492] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:09.398322 3890 ts_tablet_manager.cc:1397] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.263s user 0.098s sys 0.100s
I20260806 03:51:09.401522 3890 raft_consensus.cc:348] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:09.402511 3890 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:09.403208 3890 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:09.409119 3890 ts_tablet_manager.cc:1428] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.010s user 0.007s sys 0.002s
I20260806 03:51:09.410849 2496 tablet_copy_service.cc:342] P 608e57d72aae496aaf8613622153aa0c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-d93b8d7e5d334d3dac131b9a4c49b103 received from {username='slave'} at 127.0.0.1:56244
I20260806 03:51:09.411299 2496 tablet_copy_service.cc:433] P 608e57d72aae496aaf8613622153aa0c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-d93b8d7e5d334d3dac131b9a4c49b103 on tablet d93b8d7e5d334d3dac131b9a4c49b103 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:09.441623 3892 tablet_copy_client.cc:316] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:09.454100 3895 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:09.455250 3895 tablet_bootstrap.cc:492] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:09.456046 3895 ts_tablet_manager.cc:1397] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.077s user 0.034s sys 0.032s
I20260806 03:51:09.459031 3895 raft_consensus.cc:348] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:09.459955 3895 raft_consensus.cc:729] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:09.460651 3895 consensus_queue.cc:260] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:09.463162 3895 ts_tablet_manager.cc:1428] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260806 03:51:09.464373 2598 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:09.465062 2598 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-c210749b55e24fa0bb7b9c7cdf3cc6b0
I20260806 03:51:09.474714 2598 tablet_copy_source_session.cc:205] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:09.481866 3892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:09.491532 2598 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-2401ce53befd43c6a455e89c5df49a28 received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:09.498490 2598 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-2401ce53befd43c6a455e89c5df49a28 on tablet 2401ce53befd43c6a455e89c5df49a28 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:09.500319 3892 tablet_copy_client.cc:791] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:09.503971 3892 tablet_copy_client.cc:655] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:09.512068 3892 tablet_copy_client.cc:531] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:09.523713 3892 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:09.546595 3592 raft_consensus.cc:1203] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 03:51:09.561810 3892 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:09.563030 3892 tablet_bootstrap.cc:492] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:09.563861 3892 ts_tablet_manager.cc:1397] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.040s user 0.026s sys 0.009s
I20260806 03:51:09.566874 3892 raft_consensus.cc:348] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:09.567729 3892 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:09.568809 3892 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:09.571902 3892 ts_tablet_manager.cc:1428] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260806 03:51:09.574947 2598 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-c210749b55e24fa0bb7b9c7cdf3cc6b0 received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:09.575408 2598 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-c210749b55e24fa0bb7b9c7cdf3cc6b0 on tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:09.623968 3592 raft_consensus.cc:1203] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:09.659901 3592 raft_consensus.cc:1203] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:09.838164 3592 raft_consensus.cc:1203] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:10.064792 3862 raft_consensus.cc:1050] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:10.067775 3862 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:10.117465 3799 raft_consensus.cc:1050] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:10.121184 3799 consensus_queue.cc:237] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:10.317335 3592 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.329195 3899 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:10.338089 2372 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.346976 3923 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:51:10.382117 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 261761us
I20260806 03:51:10.385303 2578 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:51:10.412492 3593 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260806 03:51:10.427306 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 271923us
I20260806 03:51:10.434913 2484 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.453738 2372 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:51:10.483181 3906 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:10.490196 3906 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:10.501724 3899 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:10.516078 3799 raft_consensus.cc:1050] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:10.517081 3914 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:10.522856 3910 raft_consensus.cc:1050] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:10.530396 3910 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:10.519671 3799 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:10.582216 2579 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.589579 3593 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.589830 3906 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:10.595994 3906 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:51:10.599282 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 56951.2us
I20260806 03:51:10.616333 3923 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.629935 2377 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.640300 2372 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.642254 2484 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.642525 3592 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.647948 3925 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:10.655969 3926 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:10.658288 3924 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:10.664693 2478 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.670709 3910 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.675086 2572 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.686218 3906 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.709872 3594 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.715312 2368 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:10.801975 3944 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
W20260806 03:51:10.804033 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48948.9us
W20260806 03:51:10.840049 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 52145.7us
I20260806 03:51:10.835278 2235 catalog_manager.cc:5495] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.836869 2232 catalog_manager.cc:5495] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:10.843319 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 85697.2us
I20260806 03:51:10.843518 3592 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:10.896970 3949 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260806 03:51:10.908680 3924 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.914129 2369 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.921423 2479 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.938200 3591 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:10.941790 2581 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:11.100785 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 114085us
I20260806 03:51:11.088475 2372 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.318895 3594 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:11.333259 2617 debug-util.cc:398] Leaking SignalData structure 0x7b08001b7760 after lost signal to thread 735
W20260806 03:51:11.335325 2617 debug-util.cc:398] Leaking SignalData structure 0x7b08001b6f20 after lost signal to thread 2283
W20260806 03:51:11.337791 2617 debug-util.cc:398] Leaking SignalData structure 0x7b08003087a0 after lost signal to thread 2413
W20260806 03:51:11.405069 2617 debug-util.cc:398] Leaking SignalData structure 0x7b080015a440 after lost signal to thread 2516
W20260806 03:51:11.448199 2617 debug-util.cc:398] Leaking SignalData structure 0x7b08002fdee0 after lost signal to thread 2620
I20260806 03:51:11.410928 2368 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:11.456537 2617 debug-util.cc:398] Leaking SignalData structure 0x7b080030c880 after lost signal to thread 3624
W20260806 03:51:11.503223 3592 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.007s sys 0.027s
W20260806 03:51:11.503773 3592 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.007s sys 0.027s
W20260806 03:51:11.507237 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 393422us
W20260806 03:51:11.629629 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 45612.4us
I20260806 03:51:11.740007 2235 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33762 (request call id 154) took 1001 ms. Trace:
I20260806 03:51:11.742326 2232 catalog_manager.cc:5495] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:11.740492 2235 rpcz_store.cc:276] 0806 03:51:10.738445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:10.738636 (+ 191us) service_pool.cc:227] Handling call
0806 03:51:11.739932 (+1001296us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:11.577582 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:11.586635 (+ 9053us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:11.586717 (+ 82us) write_op.cc:435] Acquired schema lock
0806 03:51:11.586767 (+ 50us) tablet.cc:660] Decoding operations
0806 03:51:11.587385 (+ 618us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:11.587490 (+ 105us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:11.587563 (+ 73us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:51:11.587940 (+ 377us) tablet.cc:699] Row locks acquired
0806 03:51:11.587957 (+ 17us) write_op.cc:260] PREPARE: finished
0806 03:51:11.588272 (+ 315us) write_op.cc:270] Start()
0806 03:51:11.588474 (+ 202us) write_op.cc:276] Timestamp: P: 1785988271588185 usec, L: 0
0806 03:51:11.588493 (+ 19us) op_driver.cc:331] REPLICATION: starting
0806 03:51:11.589680 (+ 1187us) log.cc:842] Serialized 590 byte log entry
0806 03:51:11.704374 (+114694us) op_driver.cc:444] REPLICATION: finished
0806 03:51:11.716753 (+ 12379us) write_op.cc:301] APPLY: starting
0806 03:51:11.716874 (+ 121us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:11.717009 (+ 135us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:11.717167 (+ 158us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:11.717506 (+ 339us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:11.730770 (+ 13264us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:51:11.730815 (+ 45us) write_op.cc:312] APPLY: finished
0806 03:51:11.731099 (+ 284us) log.cc:842] Serialized 22 byte log entry
0806 03:51:11.737325 (+ 6226us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:11.737451 (+ 126us) write_op.cc:454] Released schema lock
0806 03:51:11.737548 (+ 97us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:11.738035 (+ 487us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:11.738063 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":11879,"num_ops":1,"prepare.queue_time_us":689215,"prepare.run_cpu_time_us":4539,"prepare.run_wall_time_us":28105,"replication_time_us":115740,"thread_start_us":4925,"threads_started":1}]]}
I20260806 03:51:11.747398 2232 catalog_manager.cc:5495] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.791785 2574 consensus_queue.cc:237] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:11.804059 2575 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:11.821574 2369 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:11.822252 3594 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:11.825513 3946 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:11.827256 3686 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:11.839323 3981 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.844195 3594 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.841962 2369 raft_consensus.cc:2932] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.865929 2479 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:11.877591 3592 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:11.879225 2377 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:11.881631 3909 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:11.883543 3912 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:11.895287 3909 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.903105 3590 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.910476 2218 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 507574ca8b05468888fa8465ccdb2f80 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:11.919091 2377 raft_consensus.cc:2932] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.947252 2479 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:11.949882 3946 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:11.962772 2217 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:11.963307 3592 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:11.963205 2235 catalog_manager.cc:5495] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:11.968526 2232 catalog_manager.cc:5495] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 4 to 5, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:11.996616 3946 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:12.011125 2472 tablet_service.cc:1514] Processing DeleteTablet for tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 with delete_type TABLET_DATA_TOMBSTONED (TS 608e57d72aae496aaf8613622153aa0c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54212
I20260806 03:51:12.024660 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet 507574ca8b05468888fa8465ccdb2f80 with delete_type TABLET_DATA_TOMBSTONED (TS fe3d17f4ad984b9088a268d58d87469c not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:44180
I20260806 03:51:12.038818 3992 tablet_replica.cc:330] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: stopping tablet replica
I20260806 03:51:12.043486 3992 raft_consensus.cc:2229] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:12.046275 3992 raft_consensus.cc:2258] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:12.067570 3992 ts_tablet_manager.cc:1905] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:12.108157 3981 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:12.115088 2484 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:12.103219 3993 tablet_replica.cc:330] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: stopping tablet replica
I20260806 03:51:12.135022 3592 raft_consensus.cc:2932] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:12.160791 2217 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2401ce53befd43c6a455e89c5df49a28 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:12.170025 2235 catalog_manager.cc:5495] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:12.189229 3993 raft_consensus.cc:2229] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:12.190308 3993 raft_consensus.cc:2258] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:12.204733 3993 ts_tablet_manager.cc:1905] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:12.239002 2359 tablet_service.cc:1514] Processing DeleteTablet for tablet 2401ce53befd43c6a455e89c5df49a28 with delete_type TABLET_DATA_TOMBSTONED (TS dff94100ffc5485f92cbe6a85addf0b2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40712
I20260806 03:51:12.256625 3992 ts_tablet_manager.cc:1918] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:12.272294 3992 log.cc:1196] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P 608e57d72aae496aaf8613622153aa0c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/wals/c210749b55e24fa0bb7b9c7cdf3cc6b0
I20260806 03:51:12.273872 2476 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:12.274796 2218 catalog_manager.cc:4841] TS 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): tablet c210749b55e24fa0bb7b9c7cdf3cc6b0 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:12.296936 3993 ts_tablet_manager.cc:1918] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260806 03:51:12.304466 3997 tablet_replica.cc:330] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: stopping tablet replica
I20260806 03:51:12.307202 3592 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:12.308598 2368 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:12.309480 3997 raft_consensus.cc:2229] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:12.309873 3909 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:12.310899 3997 raft_consensus.cc:2258] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:12.312976 3909 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:12.315163 3997 ts_tablet_manager.cc:1905] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:12.307195 3993 log.cc:1196] T 507574ca8b05468888fa8465ccdb2f80 P fe3d17f4ad984b9088a268d58d87469c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/wals/507574ca8b05468888fa8465ccdb2f80
I20260806 03:51:12.322359 2217 catalog_manager.cc:4841] TS fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): tablet 507574ca8b05468888fa8465ccdb2f80 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:12.326525 3912 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:12.328491 2368 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:12.357834 2235 catalog_manager.cc:5495] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New cstate: current_term: 1 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:12.382728 3590 raft_consensus.cc:2932] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:12.405762 3997 ts_tablet_manager.cc:1918] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:12.406603 3997 log.cc:1196] T 2401ce53befd43c6a455e89c5df49a28 P dff94100ffc5485f92cbe6a85addf0b2: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root/wals/2401ce53befd43c6a455e89c5df49a28
I20260806 03:51:12.391129 2218 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d93b8d7e5d334d3dac131b9a4c49b103 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:12.410135 2215 catalog_manager.cc:4841] TS dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): tablet 2401ce53befd43c6a455e89c5df49a28 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:12.437427 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet d93b8d7e5d334d3dac131b9a4c49b103 with delete_type TABLET_DATA_TOMBSTONED (TS fe3d17f4ad984b9088a268d58d87469c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44180
I20260806 03:51:12.438738 3993 tablet_replica.cc:330] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: stopping tablet replica
I20260806 03:51:12.476172 3993 raft_consensus.cc:2229] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:12.477246 3993 raft_consensus.cc:2258] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:12.481130 3993 ts_tablet_manager.cc:1905] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:12.498780 3993 ts_tablet_manager.cc:1918] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:12.499289 3993 log.cc:1196] T d93b8d7e5d334d3dac131b9a4c49b103 P fe3d17f4ad984b9088a268d58d87469c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/wals/d93b8d7e5d334d3dac131b9a4c49b103
I20260806 03:51:12.501308 2217 catalog_manager.cc:4841] TS fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): tablet d93b8d7e5d334d3dac131b9a4c49b103 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:13.713258 2574 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:13.726181 2482 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:13.727725 2537 consensus_peers.cc:487] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a930c23072484f7ea5974173f5843bfe. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:13.730343 2368 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:13.728483 4001 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:13.732127 3860 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:13.743860 4001 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:13.751210 2479 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:13.770018 2368 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:13.789022 2574 consensus_queue.cc:237] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:13.806919 2235 catalog_manager.cc:5495] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
W20260806 03:51:13.851248 2537 consensus_peers.cc:487] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7901535e3d4042389b2a8853a881ff1d. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:13.857802 2377 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:13.860772 3860 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:13.871773 2479 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:13.886860 3926 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:13.917508 3860 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:13.928614 2476 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:13.934309 2377 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:13.965498 2368 consensus_queue.cc:237] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:13.974697 2331 consensus_peers.cc:487] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42ea7d4af5d44cab89bf24f15376456e. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:13.976838 2476 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:13.977258 2574 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:13.978562 4008 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:13.980123 3969 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:13.992115 2236 catalog_manager.cc:5495] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:14.014284 4008 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:14.021299 2476 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:14.027127 2578 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:14.069100 2235 catalog_manager.cc:5495] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:14.075819 2578 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:14.105120 2537 consensus_peers.cc:487] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f327872ee26848bbaf04c03cc38edd54. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:14.107511 2476 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:14.109603 2368 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:14.109609 3926 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260806 03:51:14.122234 3860 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:14.137394 3860 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:14.142655 2377 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:14.158702 2476 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:14.170131 2233 catalog_manager.cc:5495] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:14.205202 4031 ts_tablet_manager.cc:927] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:14.266075 4031 tablet_copy_client.cc:316] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:14.268128 2598 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet a930c23072484f7ea5974173f5843bfe from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:14.268762 2598 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet a930c23072484f7ea5974173f5843bfe from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-a930c23072484f7ea5974173f5843bfe
I20260806 03:51:14.278911 2598 tablet_copy_source_session.cc:205] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:14.284179 4031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a930c23072484f7ea5974173f5843bfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:14.323695 4031 tablet_copy_client.cc:791] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:14.324422 4031 tablet_copy_client.cc:655] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:14.329039 4031 tablet_copy_client.cc:531] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:14.342351 4031 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:14.349502 4037 ts_tablet_manager.cc:927] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:14.359040 4037 tablet_copy_client.cc:316] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:14.371778 2598 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet 7901535e3d4042389b2a8853a881ff1d from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:14.372629 2598 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet 7901535e3d4042389b2a8853a881ff1d from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-7901535e3d4042389b2a8853a881ff1d
I20260806 03:51:14.383148 2598 tablet_copy_source_session.cc:205] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:14.385365 4031 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:14.386775 4031 tablet_bootstrap.cc:492] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:14.387888 4031 ts_tablet_manager.cc:1397] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.010s
I20260806 03:51:14.393428 4031 raft_consensus.cc:348] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:14.394450 4031 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:14.395205 4031 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:14.413012 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7901535e3d4042389b2a8853a881ff1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:14.430567 4031 ts_tablet_manager.cc:1428] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.042s user 0.007s sys 0.010s
I20260806 03:51:14.454134 2598 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-a930c23072484f7ea5974173f5843bfe received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:14.454881 2598 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-a930c23072484f7ea5974173f5843bfe on tablet a930c23072484f7ea5974173f5843bfe with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:14.466897 4037 tablet_copy_client.cc:791] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:14.467626 4037 tablet_copy_client.cc:655] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:14.472733 4037 tablet_copy_client.cc:531] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:14.495977 4037 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:14.546072 4037 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:14.549062 4037 tablet_bootstrap.cc:492] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:14.550256 4037 ts_tablet_manager.cc:1397] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.055s user 0.017s sys 0.019s
I20260806 03:51:14.559329 4037 raft_consensus.cc:348] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:14.560566 4037 raft_consensus.cc:729] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:14.561619 4037 consensus_queue.cc:260] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:14.564540 4031 ts_tablet_manager.cc:927] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:14.566145 4037 ts_tablet_manager.cc:1428] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.015s user 0.002s sys 0.009s
I20260806 03:51:14.570300 2598 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-7901535e3d4042389b2a8853a881ff1d received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:14.570878 2598 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-7901535e3d4042389b2a8853a881ff1d on tablet 7901535e3d4042389b2a8853a881ff1d with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:14.596886 3590 raft_consensus.cc:1203] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:14.606091 4031 tablet_copy_client.cc:316] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:14.618148 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet 42ea7d4af5d44cab89bf24f15376456e from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:14.619139 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet 42ea7d4af5d44cab89bf24f15376456e from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-42ea7d4af5d44cab89bf24f15376456e
I20260806 03:51:14.619767 4037 ts_tablet_manager.cc:927] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:14.648473 4037 tablet_copy_client.cc:316] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:14.650498 2598 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet f327872ee26848bbaf04c03cc38edd54 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:14.651162 2598 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet f327872ee26848bbaf04c03cc38edd54 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-f327872ee26848bbaf04c03cc38edd54
I20260806 03:51:14.663218 2598 tablet_copy_source_session.cc:205] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:14.666721 2392 tablet_copy_source_session.cc:205] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:14.667379 4037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f327872ee26848bbaf04c03cc38edd54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:14.674271 4031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42ea7d4af5d44cab89bf24f15376456e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:14.717144 4031 tablet_copy_client.cc:791] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:14.720885 4031 tablet_copy_client.cc:655] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:14.718508 4037 tablet_copy_client.cc:791] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:14.722170 4037 tablet_copy_client.cc:655] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:14.727633 4037 tablet_copy_client.cc:531] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:14.729962 4031 tablet_copy_client.cc:531] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:14.742620 4037 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:14.756680 3590 raft_consensus.cc:1203] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:14.815973 4037 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:14.817771 4037 tablet_bootstrap.cc:492] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:14.819149 4037 ts_tablet_manager.cc:1397] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.077s user 0.033s sys 0.035s
I20260806 03:51:14.821856 4031 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:14.827584 4037 raft_consensus.cc:348] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:14.829340 4037 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:14.830518 4037 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:14.835171 4037 ts_tablet_manager.cc:1428] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.016s user 0.004s sys 0.005s
I20260806 03:51:14.840792 2598 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-f327872ee26848bbaf04c03cc38edd54 received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:14.841418 2598 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-f327872ee26848bbaf04c03cc38edd54 on tablet f327872ee26848bbaf04c03cc38edd54 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:14.893224 4031 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:14.894722 4031 tablet_bootstrap.cc:492] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:14.896126 4031 ts_tablet_manager.cc:1397] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.075s user 0.032s sys 0.009s
I20260806 03:51:14.901522 4031 raft_consensus.cc:348] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:14.903090 4031 raft_consensus.cc:729] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:14.904338 4031 consensus_queue.cc:260] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:14.909008 4031 ts_tablet_manager.cc:1428] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.012s user 0.006s sys 0.004s
I20260806 03:51:14.912645 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-42ea7d4af5d44cab89bf24f15376456e received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:14.913151 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-42ea7d4af5d44cab89bf24f15376456e on tablet 42ea7d4af5d44cab89bf24f15376456e with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:15.004335 4001 raft_consensus.cc:1050] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:15.008864 4001 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:15.022010 3590 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.024214 4018 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:15.027438 2476 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.033185 2377 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.035401 3860 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:15.046391 4001 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:15.055548 3860 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.076195 3592 raft_consensus.cc:1203] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:15.081791 2368 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.078122 3590 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.110318 2233 catalog_manager.cc:5495] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:15.113998 3592 raft_consensus.cc:1203] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:15.117739 2476 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.197636 2574 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:15.215294 2476 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:15.218014 3590 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:15.225476 4001 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:15.228154 3926 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:15.241431 3860 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.254055 2476 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.261729 3592 raft_consensus.cc:2932] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.277468 2236 catalog_manager.cc:5495] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.284296 2217 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a930c23072484f7ea5974173f5843bfe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:15.289974 2359 tablet_service.cc:1514] Processing DeleteTablet for tablet a930c23072484f7ea5974173f5843bfe with delete_type TABLET_DATA_TOMBSTONED (TS dff94100ffc5485f92cbe6a85addf0b2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40712
I20260806 03:51:15.305657 4056 tablet_replica.cc:330] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: stopping tablet replica
I20260806 03:51:15.309499 4056 raft_consensus.cc:2229] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:15.310497 4056 raft_consensus.cc:2258] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:15.315362 4056 ts_tablet_manager.cc:1905] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:15.335988 4056 ts_tablet_manager.cc:1918] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:15.336560 4056 log.cc:1196] T a930c23072484f7ea5974173f5843bfe P dff94100ffc5485f92cbe6a85addf0b2: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root/wals/a930c23072484f7ea5974173f5843bfe
I20260806 03:51:15.339680 2215 catalog_manager.cc:4841] TS dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): tablet a930c23072484f7ea5974173f5843bfe (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:15.377965 4001 raft_consensus.cc:1050] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:15.384284 4001 consensus_queue.cc:237] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:15.449929 3590 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.454759 2377 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.456736 3926 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:15.457973 4018 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:15.480909 2476 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.483407 4001 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:15.498116 4018 raft_consensus.cc:1050] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:15.501252 4018 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:15.507900 3926 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.521489 2476 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.528246 3590 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.536702 2377 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.554582 2236 catalog_manager.cc:5495] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.587442 4008 raft_consensus.cc:1050] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:15.590487 4008 consensus_queue.cc:237] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:15.602566 2377 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.602566 3590 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.604540 3926 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:15.604686 4060 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:15.609514 2476 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.611466 3969 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:15.633071 2578 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.633882 4008 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.641896 3594 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.647382 4002 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:15.650180 2476 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.654495 2575 consensus_queue.cc:237] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:15.661516 2377 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:15.662669 3593 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:15.663374 3926 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:15.668465 4073 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:15.671628 3590 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:15.685031 2368 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.682617 4001 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.690649 2236 catalog_manager.cc:5495] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.697343 3860 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:15.701272 3926 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:15.705618 2575 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.718974 3926 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.725140 3590 raft_consensus.cc:2932] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.732911 2476 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.741830 2479 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.746611 2236 catalog_manager.cc:5495] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.766338 2217 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7901535e3d4042389b2a8853a881ff1d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:15.805204 2377 consensus_queue.cc:237] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:15.813758 3594 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.905651 3590 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:15.910622 2235 catalog_manager.cc:5495] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.922015 4058 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:15.926492 2476 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:15.944736 2472 tablet_service.cc:1514] Processing DeleteTablet for tablet 7901535e3d4042389b2a8853a881ff1d with delete_type TABLET_DATA_TOMBSTONED (TS 608e57d72aae496aaf8613622153aa0c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54212
I20260806 03:51:15.948650 2377 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:15.971263 4087 tablet_replica.cc:330] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: stopping tablet replica
I20260806 03:51:15.976774 4087 raft_consensus.cc:2229] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:15.978060 4087 raft_consensus.cc:2258] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:15.993266 2575 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:15.995538 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 69603.5us
I20260806 03:51:16.003913 2377 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:16.004949 3594 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:16.008451 3926 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:16.018811 4087 ts_tablet_manager.cc:1905] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:16.020211 3926 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:16.023769 4074 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:16.026592 4060 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:16.033325 3594 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:16.034313 2368 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:16.030915 3590 raft_consensus.cc:2932] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:16.064527 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 67812.2us
I20260806 03:51:16.097930 2217 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f327872ee26848bbaf04c03cc38edd54 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:16.087435 2232 catalog_manager.cc:5495] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:16.117327 4087 ts_tablet_manager.cc:1918] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:16.123831 4087 log.cc:1196] T 7901535e3d4042389b2a8853a881ff1d P 608e57d72aae496aaf8613622153aa0c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/wals/7901535e3d4042389b2a8853a881ff1d
I20260806 03:51:16.093295 2233 catalog_manager.cc:5495] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:16.126508 2218 catalog_manager.cc:4841] TS 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): tablet 7901535e3d4042389b2a8853a881ff1d (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
W20260806 03:51:16.120218 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 54803.7us
I20260806 03:51:16.128069 4090 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:16.163909 2472 tablet_service.cc:1514] Processing DeleteTablet for tablet f327872ee26848bbaf04c03cc38edd54 with delete_type TABLET_DATA_TOMBSTONED (TS 608e57d72aae496aaf8613622153aa0c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54212
I20260806 03:51:16.165961 4087 tablet_replica.cc:330] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: stopping tablet replica
I20260806 03:51:16.167202 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet 42ea7d4af5d44cab89bf24f15376456e with delete_type TABLET_DATA_TOMBSTONED (TS fe3d17f4ad984b9088a268d58d87469c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44180
I20260806 03:51:16.167935 4087 raft_consensus.cc:2229] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:16.171355 4087 raft_consensus.cc:2258] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:16.178036 4087 ts_tablet_manager.cc:1905] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:16.222968 4091 tablet_replica.cc:330] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: stopping tablet replica
I20260806 03:51:16.224001 2215 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 42ea7d4af5d44cab89bf24f15376456e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:16.232401 2484 raft_consensus.cc:2932] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:16.256192 4087 ts_tablet_manager.cc:1918] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:16.257354 4087 log.cc:1196] T f327872ee26848bbaf04c03cc38edd54 P 608e57d72aae496aaf8613622153aa0c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/wals/f327872ee26848bbaf04c03cc38edd54
I20260806 03:51:16.259979 2218 catalog_manager.cc:4841] TS 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): tablet f327872ee26848bbaf04c03cc38edd54 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:16.289261 4091 raft_consensus.cc:2229] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:16.290380 4091 raft_consensus.cc:2258] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:16.296721 4091 ts_tablet_manager.cc:1905] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:16.354245 4091 ts_tablet_manager.cc:1918] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:16.354936 4091 log.cc:1196] T 42ea7d4af5d44cab89bf24f15376456e P fe3d17f4ad984b9088a268d58d87469c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/wals/42ea7d4af5d44cab89bf24f15376456e
I20260806 03:51:16.357326 2217 catalog_manager.cc:4841] TS fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): tablet 42ea7d4af5d44cab89bf24f15376456e (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:17.420341 2575 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:17.429651 2482 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:17.429979 2537 consensus_peers.cc:487] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da90827d7f0646f5bb848777586aae32. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:17.432165 4093 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:17.440300 2369 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:17.449085 3926 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:17.472148 4073 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.474030 2479 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.492723 2379 consensus_queue.cc:237] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:17.501788 2372 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.516790 2575 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:17.516902 2331 consensus_peers.cc:487] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3ae6cf9eba9410d8307ed1da6a62ea6. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:17.519382 4097 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:17.523053 2233 catalog_manager.cc:5495] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.536909 2479 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:17.538543 4097 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:17.550416 4100 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.574302 2574 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.600071 2479 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.617348 2379 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:17.635044 2331 consensus_peers.cc:487] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2aa3fe3cb0444d6ea2df79f78a0d7543. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:17.635627 2574 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:17.637411 2479 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:17.637578 4090 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:17.639062 4097 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:17.647480 2236 catalog_manager.cc:5495] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.666373 4090 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.671978 2479 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.668605 2574 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.706512 2233 catalog_manager.cc:5495] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.719635 2379 consensus_queue.cc:237] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:17.751312 2577 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:17.761562 2331 consensus_peers.cc:487] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 940c72b441104eabbcb3ea23683268c1. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:17.763535 4060 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:17.771071 2479 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:17.774360 4100 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:17.791422 4090 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.796768 2574 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.821110 2479 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.833752 2233 catalog_manager.cc:5495] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:17.924939 4131 ts_tablet_manager.cc:927] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:17.933732 4131 tablet_copy_client.cc:316] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:17.935537 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet f3ae6cf9eba9410d8307ed1da6a62ea6 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:17.936174 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet f3ae6cf9eba9410d8307ed1da6a62ea6 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-f3ae6cf9eba9410d8307ed1da6a62ea6
I20260806 03:51:17.952260 2392 tablet_copy_source_session.cc:205] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:17.956409 4131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3ae6cf9eba9410d8307ed1da6a62ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:17.983412 4133 ts_tablet_manager.cc:927] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:17.991353 4133 tablet_copy_client.cc:316] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:17.993219 2598 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet da90827d7f0646f5bb848777586aae32 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:17.993664 4131 tablet_copy_client.cc:791] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:17.993840 2598 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet da90827d7f0646f5bb848777586aae32 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-da90827d7f0646f5bb848777586aae32
I20260806 03:51:17.994977 4131 tablet_copy_client.cc:655] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:17.999939 4131 tablet_copy_client.cc:531] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:18.003916 2598 tablet_copy_source_session.cc:205] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:18.009068 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da90827d7f0646f5bb848777586aae32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:18.015848 4131 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:18.058743 4131 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:18.059352 4133 tablet_copy_client.cc:791] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:18.059916 4131 tablet_bootstrap.cc:492] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:18.060386 4133 tablet_copy_client.cc:655] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:18.060871 4131 ts_tablet_manager.cc:1397] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.045s user 0.028s sys 0.004s
I20260806 03:51:18.066735 4133 tablet_copy_client.cc:531] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:18.068073 4131 raft_consensus.cc:348] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:18.069597 4131 raft_consensus.cc:729] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:18.070567 4131 consensus_queue.cc:260] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:18.066900 4138 ts_tablet_manager.cc:927] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:18.097901 4131 ts_tablet_manager.cc:1428] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.037s user 0.011s sys 0.001s
I20260806 03:51:18.103547 4133 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:18.108055 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-f3ae6cf9eba9410d8307ed1da6a62ea6 received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:18.108841 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-f3ae6cf9eba9410d8307ed1da6a62ea6 on tablet f3ae6cf9eba9410d8307ed1da6a62ea6 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:18.137996 4138 tablet_copy_client.cc:316] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:18.154541 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:18.155414 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-2aa3fe3cb0444d6ea2df79f78a0d7543
I20260806 03:51:18.171550 2392 tablet_copy_source_session.cc:205] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:18.175470 4138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:18.182139 4133 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:18.183317 4133 tablet_bootstrap.cc:492] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:18.184177 4133 ts_tablet_manager.cc:1397] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.081s user 0.040s sys 0.032s
I20260806 03:51:18.197973 4133 raft_consensus.cc:348] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:18.198966 4133 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:18.199667 4133 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:18.211294 4133 ts_tablet_manager.cc:1428] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260806 03:51:18.211558 4138 tablet_copy_client.cc:791] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:18.213299 2598 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-da90827d7f0646f5bb848777586aae32 received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:18.222950 2598 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-da90827d7f0646f5bb848777586aae32 on tablet da90827d7f0646f5bb848777586aae32 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:18.223171 4138 tablet_copy_client.cc:655] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:18.227342 4138 tablet_copy_client.cc:531] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:18.243996 4138 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:18.277060 4138 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:18.278699 4138 tablet_bootstrap.cc:492] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:18.279528 4138 ts_tablet_manager.cc:1397] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.019s
I20260806 03:51:18.282052 4133 ts_tablet_manager.cc:927] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:18.282581 4138 raft_consensus.cc:348] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:18.283524 4138 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:18.284216 4138 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:18.288544 4138 ts_tablet_manager.cc:1428] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260806 03:51:18.290537 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-2aa3fe3cb0444d6ea2df79f78a0d7543 received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:18.290964 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-2aa3fe3cb0444d6ea2df79f78a0d7543 on tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:18.305495 4133 tablet_copy_client.cc:316] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:18.307788 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet 940c72b441104eabbcb3ea23683268c1 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:18.308429 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet 940c72b441104eabbcb3ea23683268c1 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-940c72b441104eabbcb3ea23683268c1
I20260806 03:51:18.318074 2392 tablet_copy_source_session.cc:205] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:18.323019 4133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 940c72b441104eabbcb3ea23683268c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:18.351758 4133 tablet_copy_client.cc:791] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:18.352624 4133 tablet_copy_client.cc:655] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:18.358242 4133 tablet_copy_client.cc:531] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:18.370513 4133 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:18.381377 3590 raft_consensus.cc:1203] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:18.400621 4133 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:18.401773 4133 tablet_bootstrap.cc:492] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:18.402595 4133 ts_tablet_manager.cc:1397] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20260806 03:51:18.405890 4133 raft_consensus.cc:348] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:18.406791 4133 raft_consensus.cc:729] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:18.407502 4133 consensus_queue.cc:260] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:18.410668 4133 ts_tablet_manager.cc:1428] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.008s user 0.009s sys 0.001s
I20260806 03:51:18.413789 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-940c72b441104eabbcb3ea23683268c1 received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:18.414214 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-940c72b441104eabbcb3ea23683268c1 on tablet 940c72b441104eabbcb3ea23683268c1 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:18.600133 3594 raft_consensus.cc:1203] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260806 03:51:18.786728 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 109411us
I20260806 03:51:18.821939 3594 raft_consensus.cc:1203] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:18.822160 3593 raft_consensus.cc:1203] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20260806 03:51:18.917791 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 65228.9us
I20260806 03:51:18.949157 4160 raft_consensus.cc:1050] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:18.963928 4160 consensus_queue.cc:237] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:19.025104 3593 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.027307 4160 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.027849 2575 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.029831 4090 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.050752 2476 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.057346 4160 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.070840 4090 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.080802 3593 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.096303 2575 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.100879 2479 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.117139 2232 catalog_manager.cc:5495] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.152146 4090 raft_consensus.cc:1050] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:19.155138 4090 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:19.167245 3593 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.169291 4160 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.178861 2476 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.180570 2574 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.180680 4090 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 03:51:19.186076 4157 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.217024 4164 raft_consensus.cc:1050] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:19.222560 4164 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:19.265590 2476 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.265453 4090 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.268007 4151 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.277362 2478 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.285066 3591 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.289534 4157 raft_consensus.cc:1050] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:19.293404 3926 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 03:51:19.293880 2377 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.302249 4151 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.335352 4157 consensus_queue.cc:237] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:19.366679 2232 catalog_manager.cc:5495] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.387473 4164 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.394364 3593 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.371397 3591 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.375717 2575 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.380373 3594 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.403540 2482 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.385455 2577 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.399796 2379 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.410832 4090 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260806 03:51:19.430435 4158 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.432341 2235 catalog_manager.cc:5495] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:19.475235 2575 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:19.485087 4158 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.491425 3594 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.494956 2476 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:19.524631 2579 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.527822 4154 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:19.561844 2377 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:19.563239 3591 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:19.569018 4164 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260806 03:51:19.577790 3591 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:19.578018 2575 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:19.580466 4161 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:19.582422 4161 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:19.596253 4161 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.606796 4093 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.618103 2235 catalog_manager.cc:5495] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.635131 2575 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.634725 3594 raft_consensus.cc:2932] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.641101 2478 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:19.644608 4164 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:19.627028 3591 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.684605 2215 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:19.684314 2482 raft_consensus.cc:2932] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.688891 2476 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.721561 2217 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet da90827d7f0646f5bb848777586aae32 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:19.733559 2236 catalog_manager.cc:5495] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2) evicted. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.743091 2233 catalog_manager.cc:5495] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c reported cstate change: config changed from index 3 to 4, VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) evicted. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.752745 2379 consensus_queue.cc:237] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:19.760737 3591 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:19.760888 2476 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:19.765002 2472 tablet_service.cc:1514] Processing DeleteTablet for tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 with delete_type TABLET_DATA_TOMBSTONED (TS 608e57d72aae496aaf8613622153aa0c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:54212
I20260806 03:51:19.765720 4123 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:19.775470 2359 tablet_service.cc:1514] Processing DeleteTablet for tablet da90827d7f0646f5bb848777586aae32 with delete_type TABLET_DATA_TOMBSTONED (TS dff94100ffc5485f92cbe6a85addf0b2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40712
I20260806 03:51:19.780001 4123 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:19.792393 4158 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.797585 3594 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.799826 2476 raft_consensus.cc:2932] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.808667 4201 tablet_replica.cc:330] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: stopping tablet replica
I20260806 03:51:19.815436 4201 raft_consensus.cc:2229] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:19.818235 4201 raft_consensus.cc:2258] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:19.817219 4200 tablet_replica.cc:330] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: stopping tablet replica
I20260806 03:51:19.820492 4200 raft_consensus.cc:2229] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:19.826220 2215 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f3ae6cf9eba9410d8307ed1da6a62ea6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:19.829717 4200 raft_consensus.cc:2258] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:19.827307 2236 catalog_manager.cc:5495] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:19.834832 4201 ts_tablet_manager.cc:1905] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:19.851385 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet f3ae6cf9eba9410d8307ed1da6a62ea6 with delete_type TABLET_DATA_TOMBSTONED (TS fe3d17f4ad984b9088a268d58d87469c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44180
I20260806 03:51:19.860301 4200 ts_tablet_manager.cc:1905] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:19.885316 4202 tablet_replica.cc:330] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: stopping tablet replica
I20260806 03:51:19.886766 4202 raft_consensus.cc:2229] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:19.894480 4201 ts_tablet_manager.cc:1918] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:19.895124 4201 log.cc:1196] T da90827d7f0646f5bb848777586aae32 P dff94100ffc5485f92cbe6a85addf0b2: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-0-root/wals/da90827d7f0646f5bb848777586aae32
I20260806 03:51:19.904160 4202 raft_consensus.cc:2258] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:19.907119 2215 catalog_manager.cc:4841] TS dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): tablet da90827d7f0646f5bb848777586aae32 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:19.908114 4202 ts_tablet_manager.cc:1905] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:19.921940 4200 ts_tablet_manager.cc:1918] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:19.922554 4200 log.cc:1196] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P 608e57d72aae496aaf8613622153aa0c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-1-root/wals/2aa3fe3cb0444d6ea2df79f78a0d7543
I20260806 03:51:19.937283 2218 catalog_manager.cc:4841] TS 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): tablet 2aa3fe3cb0444d6ea2df79f78a0d7543 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:19.946049 4202 ts_tablet_manager.cc:1918] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:19.946700 4202 log.cc:1196] T f3ae6cf9eba9410d8307ed1da6a62ea6 P fe3d17f4ad984b9088a268d58d87469c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/wals/f3ae6cf9eba9410d8307ed1da6a62ea6
I20260806 03:51:19.948654 2217 catalog_manager.cc:4841] TS fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): tablet f3ae6cf9eba9410d8307ed1da6a62ea6 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:20.063964 2379 consensus_queue.cc:237] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:20.072068 3591 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:20.073796 4158 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:20.075431 2476 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260806 03:51:20.084949 4159 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:20.089725 3591 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:20.091274 4158 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:51:20.106206 2215 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 940c72b441104eabbcb3ea23683268c1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:51:20.110445 2482 raft_consensus.cc:2932] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:20.122031 2235 catalog_manager.cc:5495] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 3 to 4, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:20.142932 2568 tablet_service.cc:1514] Processing DeleteTablet for tablet 940c72b441104eabbcb3ea23683268c1 with delete_type TABLET_DATA_TOMBSTONED (TS fe3d17f4ad984b9088a268d58d87469c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:44180
I20260806 03:51:20.144057 4202 tablet_replica.cc:330] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: stopping tablet replica
I20260806 03:51:20.145152 4202 raft_consensus.cc:2229] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 03:51:20.145987 4202 raft_consensus.cc:2258] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 03:51:20.149585 4202 ts_tablet_manager.cc:1905] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260806 03:51:20.171798 4202 ts_tablet_manager.cc:1918] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260806 03:51:20.172363 4202 log.cc:1196] T 940c72b441104eabbcb3ea23683268c1 P fe3d17f4ad984b9088a268d58d87469c: Deleting WAL directory at /tmp/dist-test-taskotlWEo/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785988177119491-732-0/minicluster-data/ts-2-root/wals/940c72b441104eabbcb3ea23683268c1
I20260806 03:51:20.173816 2217 catalog_manager.cc:4841] TS fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): tablet 940c72b441104eabbcb3ea23683268c1 (table test-workload [id=9c22b399a4bb4dc19ff8a892da9b81fd]) successfully deleted
I20260806 03:51:20.502562 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:20.565788 2377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:20.566481 2377 raft_consensus.cc:595] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:20.581457 2579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:51:20.582139 2579 raft_consensus.cc:595] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:20.584921 2377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:20.585631 2377 raft_consensus.cc:595] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:20.588265 2377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dad99dac270543b19675707d75149e78"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:20.588764 2377 raft_consensus.cc:595] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:20.591113 2377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:20.591723 2377 raft_consensus.cc:595] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:20.594075 2377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:20.594624 2377 raft_consensus.cc:595] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:20.595678 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 03:51:20.597817 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:51:20.598518 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:20.732200 4209 raft_consensus.cc:979] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:20.732877 4158 raft_consensus.cc:1067] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:20.734575 2478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dad99dac270543b19675707d75149e78"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54234
I20260806 03:51:20.735185 2478 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:20.735605 2478 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.742483 2478 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:20.744884 2478 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:20.745901 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:20.746001 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:20.746927 2574 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.746872 2379 raft_consensus.cc:3032] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:20.747404 2379 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:20.748297 2379 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:20.749678 2379 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.755084 2574 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:20.757054 2436 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:20.758484 4198 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:20.759006 4198 raft_consensus.cc:686] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:20.759968 4198 consensus_queue.cc:237] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:20.764765 2379 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:20.779374 4154 raft_consensus.cc:979] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:20.780243 4209 raft_consensus.cc:1067] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:20.781961 2478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54234
I20260806 03:51:20.782579 2478 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:20.782949 2478 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.781253 2235 catalog_manager.cc:5495] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:20.799497 2478 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:20.801997 2478 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:20.802994 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:20.803733 2574 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.803021 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:20.805684 2379 raft_consensus.cc:3032] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:20.806469 2379 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:20.807471 2379 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:20.818948 2574 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:20.823648 2436 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:20.825384 4198 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:20.809111 2379 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.826196 4198 raft_consensus.cc:686] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:20.828260 4198 consensus_queue.cc:237] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:20.842727 2379 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:20.852360 2235 catalog_manager.cc:5495] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:20.880100 4158 raft_consensus.cc:979] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:20.880774 4209 raft_consensus.cc:1067] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:20.882438 3591 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3"
dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:36468
I20260806 03:51:20.883057 3591 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:20.883414 3591 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.891709 3591 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:20.895726 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:20.895948 3591 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:20.896538 2377 raft_consensus.cc:3032] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:20.896896 2377 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:20.897109 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:20.897784 2574 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.897652 2377 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:20.899124 2377 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.908537 2574 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:20.908998 2377 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:20.909879 3545 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:20.911679 4216 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:20.912914 4216 raft_consensus.cc:686] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:20.914192 4216 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:20.924851 2235 catalog_manager.cc:5495] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 2 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:20.947181 4159 raft_consensus.cc:979] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:20.948299 4209 raft_consensus.cc:1067] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:20.950663 3591 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940"
dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:36468
I20260806 03:51:20.951447 3591 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:20.951845 3591 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.962988 3591 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:20.966140 3591 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:20.966256 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:20.966459 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:20.967013 2579 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.967130 2377 raft_consensus.cc:3032] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:20.967511 2377 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:20.968513 2377 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:20.969960 2377 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:20.977823 2377 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:20.984526 3542 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:20.985692 4216 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:20.986274 4216 raft_consensus.cc:686] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:20.987354 4216 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:20.990474 2579 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:20.991501 4158 raft_consensus.cc:979] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:20.992105 4158 raft_consensus.cc:1067] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:20.994060 3591 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1"
dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:36468
I20260806 03:51:20.994694 3591 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:20.995087 3591 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.003800 3591 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:21.011375 3591 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:21.014180 2235 catalog_manager.cc:5495] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 2 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:21.017246 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:21.017524 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:21.017935 2377 raft_consensus.cc:3032] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:21.018226 2478 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.018302 2377 raft_consensus.cc:729] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:21.019300 2377 consensus_queue.cc:260] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:21.020493 2377 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.028240 2377 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:21.028270 2478 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:21.030263 3542 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:21.031225 4216 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:21.031687 4216 raft_consensus.cc:686] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:21.032747 4216 consensus_queue.cc:237] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:21.040515 4164 raft_consensus.cc:979] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:21.041049 4214 raft_consensus.cc:1067] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:21.040628 2235 catalog_manager.cc:5495] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 2 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:21.042503 2478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54220
I20260806 03:51:21.042976 2478 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:21.043318 2478 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.052378 2478 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.054728 2478 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:21.056496 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:21.056770 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:21.057149 2377 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.057396 2579 raft_consensus.cc:3032] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:21.057721 2579 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:21.058337 2579 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.060220 2579 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.065567 2377 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:21.066794 2429 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:21.067593 4203 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:21.067482 2579 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:21.068279 4203 raft_consensus.cc:686] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:21.069258 4203 consensus_queue.cc:237] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.080873 2235 catalog_manager.cc:5495] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
W20260806 03:51:21.201025 2302 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20260806 03:51:21.205658 2302 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 01b082125feb462d9ba064ed2acf4dfd, TS dff94100ffc5485f92cbe6a85addf0b2 -> TS cf6042e85bef47c4aac9d064d91501d1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:21.206097 2302 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 01b082125feb462d9ba064ed2acf4dfd, TS dff94100ffc5485f92cbe6a85addf0b2 -> TS cf6042e85bef47c4aac9d064d91501d1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:21.210178 2302 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8390916b6ac84b59b3943bf835a36a4c, TS fe3d17f4ad984b9088a268d58d87469c -> TS cf6042e85bef47c4aac9d064d91501d1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:21.210565 2302 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8390916b6ac84b59b3943bf835a36a4c, TS fe3d17f4ad984b9088a268d58d87469c -> TS cf6042e85bef47c4aac9d064d91501d1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:21.214367 2302 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 2ccb469db1664d3e941e7a360d048af2, TS 608e57d72aae496aaf8613622153aa0c -> TS cf6042e85bef47c4aac9d064d91501d1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:21.214659 2302 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 2ccb469db1664d3e941e7a360d048af2, TS 608e57d72aae496aaf8613622153aa0c -> TS cf6042e85bef47c4aac9d064d91501d1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:21.218608 2302 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet ac682b8be46c4b6599610372a4c8c727, TS dff94100ffc5485f92cbe6a85addf0b2 -> TS cf6042e85bef47c4aac9d064d91501d1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:21.218935 2302 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet ac682b8be46c4b6599610372a4c8c727, TS dff94100ffc5485f92cbe6a85addf0b2 -> TS cf6042e85bef47c4aac9d064d91501d1 move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.224088 2379 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:21.224749 2579 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:21.225917 4203 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:21.226964 2579 raft_consensus.cc:1261] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:21.227309 4203 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:21.228811 4203 consensus_queue.cc:1035] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:21.288491 2379 raft_consensus.cc:1261] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:21.309486 4233 consensus_queue.cc:1035] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260806 03:51:21.322710 2574 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:21.324570 4216 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:51:21.348595 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 65192.5us
I20260806 03:51:21.410218 2379 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:21.413657 4218 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:21.460453 2379 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:21.462307 4218 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:51:21.477923 2377 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:21.480129 4216 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:51:21.480551 2574 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:21.482065 4216 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260806 03:51:21.558463 2482 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:21.559944 4257 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:21.576005 2574 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:21.577796 4229 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:21.590049 2369 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:21.592772 4234 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:21.645792 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:21.688740 2369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:21.689321 2369 raft_consensus.cc:595] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:21.691668 2369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:21.692222 2369 raft_consensus.cc:595] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:21.694402 2574 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:51:21.694836 2574 raft_consensus.cc:595] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:21.697149 2369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8655b80017d7422082efde215e797b5b"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:21.697624 2369 raft_consensus.cc:595] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:21.699640 2369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:21.700148 2369 raft_consensus.cc:595] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:21.702090 2369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:21.702518 2369 raft_consensus.cc:595] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:21.703353 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 03:51:21.704993 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:51:21.705647 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:21.789968 4159 raft_consensus.cc:979] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:21.790764 4243 raft_consensus.cc:1067] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:21.792310 2482 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54234
I20260806 03:51:21.792858 2482 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:21.793133 2482 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.801231 2482 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.803015 2482 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:21.804160 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:21.804909 2574 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.804958 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:21.805588 2369 raft_consensus.cc:3032] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:21.806416 2369 raft_consensus.cc:729] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:21.807600 2369 consensus_queue.cc:260] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.809031 2369 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.812809 2574 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:21.814069 2436 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:21.814777 4229 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:21.815217 4229 raft_consensus.cc:686] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:21.815794 2369 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:21.816232 4229 consensus_queue.cc:237] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.827092 2232 catalog_manager.cc:5495] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:21.875831 4243 raft_consensus.cc:979] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:21.876502 4159 raft_consensus.cc:1067] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:21.878096 3594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e"
dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:36468
I20260806 03:51:21.878701 3594 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:21.879122 3594 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.886852 3594 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:21.888952 3594 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:21.890271 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:21.890985 2482 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.891304 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:21.891829 2369 raft_consensus.cc:3032] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:21.892140 2369 raft_consensus.cc:729] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:21.892932 2369 consensus_queue.cc:260] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:21.894281 2369 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.897821 2482 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:21.899474 3547 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:21.900665 4257 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:21.901039 2369 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:21.901202 4257 raft_consensus.cc:686] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:21.902315 4257 consensus_queue.cc:237] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:21.913815 2232 catalog_manager.cc:5495] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 2 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:21.939435 4214 raft_consensus.cc:979] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:21.940157 4271 raft_consensus.cc:1067] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Signalling peer dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:21.941686 2369 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:40728
I20260806 03:51:21.942193 2369 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:21.942502 2369 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.949978 2369 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.951669 2369 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:21.952987 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:21.953670 2482 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.953589 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:21.954351 2574 raft_consensus.cc:3032] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:21.954653 2574 raft_consensus.cc:729] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:21.956168 2574 consensus_queue.cc:260] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.957460 2574 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.963472 2482 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:21.964668 2332 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:21.965672 4159 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:21.966113 4159 raft_consensus.cc:686] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:21.966504 2574 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:21.967062 4159 consensus_queue.cc:237] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:21.988309 4243 raft_consensus.cc:979] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:21.988893 4159 raft_consensus.cc:1067] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:21.990358 2482 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8655b80017d7422082efde215e797b5b"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54234
I20260806 03:51:21.990970 2482 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:21.991339 2482 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:21.996009 4239 raft_consensus.cc:979] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:21.996652 4239 raft_consensus.cc:1067] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:21.998581 2478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54234
I20260806 03:51:22.000573 2478 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:22.000939 2478 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:21.997781 2232 catalog_manager.cc:5495] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 2 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:22.003662 2482 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.005883 2482 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:22.007104 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:22.008351 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:22.008999 2372 raft_consensus.cc:3032] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:22.009322 2372 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:22.009490 2478 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.010116 2372 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.011454 2478 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:22.012692 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:22.013316 2369 raft_consensus.cc:3032] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:22.013609 2369 raft_consensus.cc:729] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:22.011490 2372 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:22.014312 2369 consensus_queue.cc:260] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.015302 2369 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:22.007769 2574 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:22.017892 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:22.018525 2579 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:22.022759 2372 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:22.022998 2369 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:22.023398 2574 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:22.024926 2429 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:22.025731 4234 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:22.026221 4234 raft_consensus.cc:686] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:22.026572 2579 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:22.027099 4234 consensus_queue.cc:237] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.027900 2429 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:22.029011 4236 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:22.029495 4236 raft_consensus.cc:686] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:22.030375 4236 consensus_queue.cc:237] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.052762 2232 catalog_manager.cc:5495] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:22.054273 2232 catalog_manager.cc:5495] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 3, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 3 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:22.157176 4243 raft_consensus.cc:979] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:22.157851 4159 raft_consensus.cc:1067] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:22.159638 2478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54234
I20260806 03:51:22.160178 2478 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:22.160485 2478 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:22.167620 2478 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.169260 2478 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:22.170218 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:22.170367 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:22.170856 2369 raft_consensus.cc:3032] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:22.171010 2579 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:22.171150 2369 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:22.172092 2369 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.173192 2369 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:22.177188 2579 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:22.178361 2436 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:22.179152 4229 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:22.179618 2369 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:22.179735 4229 raft_consensus.cc:686] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:22.181883 4229 consensus_queue.cc:237] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:22.193266 2232 catalog_manager.cc:5495] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:22.225800 2369 raft_consensus.cc:1261] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:22.227270 4229 consensus_queue.cc:1035] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:22.244385 2579 raft_consensus.cc:1261] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:22.246217 4236 consensus_queue.cc:1035] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:22.317847 2579 consensus_queue.cc:237] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:22.324707 2369 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:22.324692 2478 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:22.325400 2537 consensus_peers.cc:487] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78ff87b2196e4600aed5d8442f9b9bdd. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:22.326887 4214 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:22.328158 4271 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:22.342615 4271 raft_consensus.cc:2932] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.343350 2478 raft_consensus.cc:2932] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.347353 2369 raft_consensus.cc:2932] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.364125 2372 consensus_queue.cc:237] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:22.369475 2232 catalog_manager.cc:5495] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 03:51:22.381311 2331 consensus_peers.cc:487] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfa9a0499fc04e10aefa0602a7adbbb1. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:22.381553 2478 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:22.383347 4239 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:22.384007 2579 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:22.384316 2478 raft_consensus.cc:1261] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:22.387028 4239 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:22.387817 4159 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:22.433543 4239 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.438531 2476 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:22.439437 2579 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.446834 2476 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.449159 4257 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:51:22.451290 2579 raft_consensus.cc:1261] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:22.453207 4159 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:22.480336 2235 catalog_manager.cc:5495] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.499423 2574 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:22.504041 2574 consensus_queue.cc:237] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:22.516599 4236 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:22.536123 2369 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:22.539778 4236 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:22.554085 2369 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 03:51:22.557749 2372 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:22.556042 4271 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260806 03:51:22.562151 2372 raft_consensus.cc:1261] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:22.566298 2478 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 03:51:22.571376 2537 consensus_peers.cc:487] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14fa621f65da481d96d569096ded0a35. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:22.592389 4236 consensus_queue.cc:1035] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:22.595981 4302 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:51:22.607518 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 122143us
I20260806 03:51:22.617184 2578 raft_consensus.cc:1261] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:22.623168 4276 consensus_queue.cc:1035] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 03:51:22.644009 4164 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:22.646004 2578 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:22.647670 4234 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:22.711136 4164 raft_consensus.cc:2932] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.756209 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:22.860580 2372 raft_consensus.cc:2932] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:22.906726 2372 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:22.911266 4315 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 03:51:22.966050 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 94348.8us
I20260806 03:51:23.019773 4346 ts_tablet_manager.cc:927] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:23.031352 2482 consensus_queue.cc:237] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:23.128501 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 45056.3us
I20260806 03:51:23.145597 2235 catalog_manager.cc:5495] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:23.190544 2483 raft_consensus.cc:2932] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:23.270916 4352 ts_tablet_manager.cc:927] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:23.341498 4346 tablet_copy_client.cc:316] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.1:39643
I20260806 03:51:23.390937 4352 tablet_copy_client.cc:316] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:23.417897 4366 ts_tablet_manager.cc:927] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:23.423353 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 94700.4us
I20260806 03:51:23.426704 2392 tablet_copy_service.cc:140] P dff94100ffc5485f92cbe6a85addf0b2: Received BeginTabletCopySession request for tablet dfa9a0499fc04e10aefa0602a7adbbb1 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:37370)
I20260806 03:51:23.427385 2392 tablet_copy_service.cc:161] P dff94100ffc5485f92cbe6a85addf0b2: Beginning new tablet copy session on tablet dfa9a0499fc04e10aefa0602a7adbbb1 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:37370: session id = cf6042e85bef47c4aac9d064d91501d1-dfa9a0499fc04e10aefa0602a7adbbb1
I20260806 03:51:23.439348 2372 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:51:23.442653 4307 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:51:23.450493 2436 consensus_peers.cc:487] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c -> Peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Couldn't send request to peer cf6042e85bef47c4aac9d064d91501d1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47ed8dc77b144ba08169a0090b787536. This is attempt 1: this message will repeat every 5th retry.
I20260806 03:51:23.484735 2575 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260806 03:51:23.506675 4366 tablet_copy_client.cc:316] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.3:46407
I20260806 03:51:23.534436 2392 tablet_copy_source_session.cc:205] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Tablet Copy: opened 0 blocks and 1 log segments
W20260806 03:51:23.713737 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 217906us
I20260806 03:51:23.708899 2369 raft_consensus.cc:2932] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:23.708938 4346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa9a0499fc04e10aefa0602a7adbbb1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260806 03:51:23.775355 2278 debug-util.cc:398] Leaking SignalData structure 0x7b0800034a20 after lost signal to thread 735
W20260806 03:51:23.776990 2278 debug-util.cc:398] Leaking SignalData structure 0x7b080003e700 after lost signal to thread 2283
W20260806 03:51:23.789758 2278 debug-util.cc:398] Leaking SignalData structure 0x7b0800127580 after lost signal to thread 2413
W20260806 03:51:23.793598 2278 debug-util.cc:398] Leaking SignalData structure 0x7b080031ffa0 after lost signal to thread 2516
I20260806 03:51:23.648710 4307 raft_consensus.cc:2932] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
W20260806 03:51:23.812615 2278 debug-util.cc:398] Leaking SignalData structure 0x7b080031df20 after lost signal to thread 2620
W20260806 03:51:23.825817 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 54363.6us
W20260806 03:51:23.830729 2278 debug-util.cc:398] Leaking SignalData structure 0x7b0800312a40 after lost signal to thread 3624
I20260806 03:51:23.869360 4234 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:51:23.905419 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 136534us
I20260806 03:51:24.001739 2598 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet 78ff87b2196e4600aed5d8442f9b9bdd from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:24.002203 2597 tablet_copy_service.cc:140] P fe3d17f4ad984b9088a268d58d87469c: Received BeginTabletCopySession request for tablet 14fa621f65da481d96d569096ded0a35 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56018)
I20260806 03:51:24.001885 2483 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "fe3d17f4ad984b9088a268d58d87469c"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:51:24.008832 2483 raft_consensus.cc:595] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Received request to transfer leadership to TS fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:24.002951 2598 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet 78ff87b2196e4600aed5d8442f9b9bdd from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-78ff87b2196e4600aed5d8442f9b9bdd
I20260806 03:51:24.015882 2597 tablet_copy_service.cc:161] P fe3d17f4ad984b9088a268d58d87469c: Beginning new tablet copy session on tablet 14fa621f65da481d96d569096ded0a35 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56018: session id = cf6042e85bef47c4aac9d064d91501d1-14fa621f65da481d96d569096ded0a35
W20260806 03:51:24.020664 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 75674.6us
W20260806 03:51:24.048322 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 76454.2us
W20260806 03:51:24.049319 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 182529us
I20260806 03:51:24.066372 2480 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:51:24.067304 2480 raft_consensus.cc:595] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:24.077862 2373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:24.079135 2373 raft_consensus.cc:595] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:24.059309 4307 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:56228 (request call id 6) took 1031 ms. Trace:
I20260806 03:51:24.095058 4307 rpcz_store.cc:276] 0806 03:51:23.019235 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:23.029093 (+ 9858us) service_pool.cc:227] Handling call
0806 03:51:23.074104 (+ 45011us) log.cc:842] Serialized 492 byte log entry
0806 03:51:23.431906 (+357802us) spinlock_profiling.cc:243] Waited 886 us on lock 0x7b18001746a0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b144ca 00007f0d831f5f99 00007f0d799d8990 00007f0d79b9a864 00007f0d79b9a155 00007f0d7ca3f320 00007f0d7caa5113 00007f0d7cacb7e2 00007f0d81a07f39
0806 03:51:23.445328 (+ 13422us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b18001746a0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d764 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:51:24.050800 (+605472us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":221,"mutex_wait_us":21601,"raft.queue_time_us":60543,"raft.run_cpu_time_us":2238,"raft.run_wall_time_us":36245,"spinlock_wait_cycles":6987648,"thread_start_us":298917,"threads_started":1,"wal-append.queue_time_us":349123}
I20260806 03:51:24.134836 2232 catalog_manager.cc:5495] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:24.153764 4391 ts_tablet_manager.cc:927] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:24.168282 2598 tablet_copy_source_session.cc:205] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:24.190315 2575 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:51:24.205327 2575 raft_consensus.cc:595] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:24.214249 2482 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:51:24.219545 2482 raft_consensus.cc:595] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:24.223762 4352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78ff87b2196e4600aed5d8442f9b9bdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:24.171375 4346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:36468 (request call id 237) took 1175 ms. Trace:
I20260806 03:51:24.242369 2483 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dad99dac270543b19675707d75149e78"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:51:24.251139 2483 raft_consensus.cc:595] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:24.168344 4391 tablet_copy_client.cc:316] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.183.2:45689
I20260806 03:51:24.254556 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260806 03:51:24.256703 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:51:24.257606 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:24.258249 2496 tablet_copy_service.cc:140] P 608e57d72aae496aaf8613622153aa0c: Received BeginTabletCopySession request for tablet 47ed8dc77b144ba08169a0090b787536 from peer cf6042e85bef47c4aac9d064d91501d1 ({username='slave'} at 127.0.0.1:56244)
I20260806 03:51:24.258970 2496 tablet_copy_service.cc:161] P 608e57d72aae496aaf8613622153aa0c: Beginning new tablet copy session on tablet 47ed8dc77b144ba08169a0090b787536 from peer cf6042e85bef47c4aac9d064d91501d1 at {username='slave'} at 127.0.0.1:56244: session id = cf6042e85bef47c4aac9d064d91501d1-47ed8dc77b144ba08169a0090b787536
I20260806 03:51:24.249321 2597 tablet_copy_source_session.cc:205] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:24.249320 4346 rpcz_store.cc:276] 0806 03:51:22.995780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:22.995942 (+ 162us) service_pool.cc:227] Handling call
0806 03:51:23.020172 (+ 24230us) ts_tablet_manager.cc:928] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
0806 03:51:23.700895 (+680723us) tablet_copy_client.cc:361] Tablet copy session begun
0806 03:51:24.142416 (+441521us) tablet_metadata.cc:641] Metadata flushed
0806 03:51:24.142879 (+ 463us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 03:51:24.169912 (+ 27033us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 03:51:24.170538 (+ 626us) tablet_replica.cc:205] Creating consensus instance
0806 03:51:24.171078 (+ 540us) ts_tablet_manager.cc:983] Replica registered.
0806 03:51:24.171219 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":14592,"tablet-copy.queue_time_us":22236,"thread_start_us":324118,"threads_started":2}
I20260806 03:51:24.310037 4346 tablet_copy_client.cc:791] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:24.310886 4346 tablet_copy_client.cc:655] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:24.323891 4346 tablet_copy_client.cc:531] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:24.307989 2496 tablet_copy_source_session.cc:205] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 03:51:24.329315 4307 raft_consensus.cc:979] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: : Instructing follower cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:24.333705 4307 raft_consensus.cc:1067] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Signalling peer cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:24.336686 4391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47ed8dc77b144ba08169a0090b787536. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:24.359319 4366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14fa621f65da481d96d569096ded0a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 03:51:24.367832 4385 raft_consensus.cc:979] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:24.367897 3594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80"
dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:36484
I20260806 03:51:24.368964 3594 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:24.369381 3594 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.369282 4236 raft_consensus.cc:979] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: : Instructing follower fe3d17f4ad984b9088a268d58d87469c to start an election
I20260806 03:51:24.371876 4236 raft_consensus.cc:1067] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Signalling peer fe3d17f4ad984b9088a268d58d87469c to start an election
I20260806 03:51:24.384308 4385 raft_consensus.cc:1067] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:24.314697 2577 raft_consensus.cc:2932] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
I20260806 03:51:24.403105 4315 raft_consensus.cc:979] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: : Instructing follower dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:24.403839 4236 raft_consensus.cc:1067] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Signalling peer dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:24.409621 3594 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:24.412845 2368 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dad99dac270543b19675707d75149e78"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:40734
I20260806 03:51:24.413076 2579 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
from {username='slave'} at 127.0.0.1:44204
I20260806 03:51:24.413538 2368 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:24.413955 2368 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.422518 4307 raft_consensus.cc:979] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: : Instructing follower cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:24.413894 2579 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:24.423873 2579 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.428946 4307 raft_consensus.cc:1067] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Signalling peer cf6042e85bef47c4aac9d064d91501d1 to start an election
I20260806 03:51:24.437954 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:24.438913 2370 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.444271 3594 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:24.453011 2476 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54234
I20260806 03:51:24.453630 2476 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:24.453994 2476 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:24.455864 3594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103"
dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:36484
I20260806 03:51:24.456585 3594 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:24.457667 3594 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:24.474509 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:24.491230 2482 raft_consensus.cc:3032] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:24.491613 2482 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:24.492507 2482 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:24.512094 2579 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:24.516564 2579 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:24.517011 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:24.517693 2377 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.520972 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:24.521659 2478 raft_consensus.cc:3032] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:24.521955 2478 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:24.522698 2478 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:24.523617 2377 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:24.523860 2478 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.532099 2476 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:24.532899 3594 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:24.537808 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:24.538754 2373 raft_consensus.cc:3032] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:24.539171 2373 raft_consensus.cc:729] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:24.541127 2476 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:24.543062 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:24.544706 3594 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:24.545815 2579 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:24.546152 2534 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:24.547022 2482 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.549392 2478 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:24.550704 2373 consensus_queue.cc:260] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:24.569371 4379 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:24.569741 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:24.587308 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:24.588177 2376 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:24.572721 4379 raft_consensus.cc:686] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:24.593827 2368 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:24.585152 2477 raft_consensus.cc:3032] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:24.631397 2482 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:24.631433 2368 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:24.628777 2477 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:24.642480 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:24.647272 2480 raft_consensus.cc:3032] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:24.636008 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:24.648612 2576 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.647760 2480 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:24.652330 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:24.652966 2480 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:24.657152 2480 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:24.673002 2579 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:24.639523 2477 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:24.665189 4257 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:24.675779 2477 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:24.666968 4164 raft_consensus.cc:979] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:24.678921 4164 raft_consensus.cc:1067] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:24.693030 2436 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:24.707068 4229 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:24.638540 4379 consensus_queue.cc:237] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:24.716233 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 65154.9us
I20260806 03:51:24.760344 4346 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:24.727044 2373 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:24.767983 2376 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:24.791132 2373 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:24.796612 2478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54220
I20260806 03:51:24.797448 2478 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:24.797834 2478 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:24.807082 2478 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:24.813519 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:24.814375 2376 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:24.816457 2572 raft_consensus.cc:1226] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:24.826176 2576 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:24.827148 2578 raft_consensus.cc:1226] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:24.827839 2376 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:24.845424 2477 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:24.847081 3547 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:24.847471 2429 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:24.792398 4257 raft_consensus.cc:686] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:24.848501 4408 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:24.848625 4307 raft_consensus.cc:2781] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:24.849390 4408 raft_consensus.cc:686] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:24.849416 4307 raft_consensus.cc:686] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:24.849798 4257 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:24.856336 2480 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:24.850960 4408 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:24.859542 2478 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:24.862996 2332 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:24.809340 4229 raft_consensus.cc:686] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:24.850917 4307 consensus_queue.cc:237] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:24.883657 4352 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:36474 (request call id 196) took 1853 ms. Trace:
I20260806 03:51:24.876415 4331 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:24.884876 4331 raft_consensus.cc:686] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:24.884445 4352 rpcz_store.cc:276] 0806 03:51:23.029821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:23.030228 (+ 407us) service_pool.cc:227] Handling call
0806 03:51:23.271636 (+241408us) ts_tablet_manager.cc:928] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
0806 03:51:24.221786 (+950150us) tablet_copy_client.cc:361] Tablet copy session begun
0806 03:51:24.469365 (+247579us) tablet_metadata.cc:641] Metadata flushed
0806 03:51:24.505695 (+ 36330us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 03:51:24.835509 (+329814us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 03:51:24.836002 (+ 493us) tablet_replica.cc:205] Creating consensus instance
0806 03:51:24.883506 (+ 47504us) ts_tablet_manager.cc:983] Replica registered.
0806 03:51:24.883589 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":440,"tablet-copy.queue_time_us":238871,"thread_start_us":301954,"threads_started":2}
I20260806 03:51:24.886962 4352 tablet_copy_client.cc:791] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:24.887704 4352 tablet_copy_client.cc:655] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:24.871838 4229 consensus_queue.cc:237] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:24.949697 2370 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.510s user 0.014s sys 0.007s
I20260806 03:51:24.950453 2370 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:24.966200 2376 raft_consensus.cc:1226] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:24.886646 4331 consensus_queue.cc:237] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:25.009123 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:25.010442 2579 raft_consensus.cc:3032] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:25.010980 2579 raft_consensus.cc:729] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:25.012398 2579 consensus_queue.cc:260] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:25.014132 2579 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:25.023380 2480 raft_consensus.cc:1226] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:25.021443 4374 consensus_queue.cc:1159] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "dff94100ffc5485f92cbe6a85addf0b2" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:51:25.040282 2232 catalog_manager.cc:5495] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 2 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:51:25.062067 2579 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:25.072597 4391 tablet_copy_client.cc:791] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:25.098987 4352 tablet_copy_client.cc:531] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:25.102100 4391 tablet_copy_client.cc:655] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:25.126715 2235 catalog_manager.cc:5495] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:25.144266 2233 catalog_manager.cc:5495] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 2 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 3 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
W20260806 03:51:25.164780 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 44299.2us
I20260806 03:51:25.169345 4366 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.StartTabletCopy from 127.0.0.1:36474 (request call id 198) took 1877 ms. Trace:
I20260806 03:51:25.170071 4366 rpcz_store.cc:276] 0806 03:51:23.292111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:23.292308 (+ 197us) service_pool.cc:227] Handling call
0806 03:51:23.423484 (+131176us) ts_tablet_manager.cc:928] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
0806 03:51:24.356974 (+933490us) tablet_copy_client.cc:361] Tablet copy session begun
0806 03:51:24.752745 (+395771us) tablet_metadata.cc:641] Metadata flushed
0806 03:51:24.753097 (+ 352us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 03:51:25.124687 (+371590us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 03:51:25.168251 (+ 43564us) tablet_replica.cc:205] Creating consensus instance
0806 03:51:25.169075 (+ 824us) ts_tablet_manager.cc:983] Replica registered.
0806 03:51:25.169235 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":367,"tablet-copy.queue_time_us":124633,"thread_start_us":159102,"threads_started":2}
W20260806 03:51:25.181733 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 81371.6us
I20260806 03:51:25.206209 4391 tablet_copy_client.cc:531] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 03:51:25.185153 4366 tablet_copy_client.cc:791] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 0 data blocks...
I20260806 03:51:25.210240 4366 tablet_copy_client.cc:655] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Starting download of 1 WAL segments...
I20260806 03:51:25.213037 4352 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:25.224807 2236 catalog_manager.cc:5495] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 2 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 3 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:25.255829 4366 tablet_copy_client.cc:531] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260806 03:51:25.271952 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 153576us
I20260806 03:51:25.275720 2370 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:25.278640 4391 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:25.288201 4388 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:25.305231 2370 raft_consensus.cc:1261] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:25.307896 4375 consensus_queue.cc:1035] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 03:51:25.355051 4346 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:25.357267 4346 tablet_bootstrap.cc:492] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:25.358388 2477 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:25.362126 4409 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:25.365981 2370 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:25.366546 2578 raft_consensus.cc:1261] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:25.366698 4346 ts_tablet_manager.cc:1397] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.676s user 0.033s sys 0.029s
I20260806 03:51:25.370759 4236 consensus_queue.cc:1035] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:25.371644 2581 raft_consensus.cc:1261] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:25.373131 4364 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:51:25.388746 2370 raft_consensus.cc:1261] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:25.390033 4307 consensus_queue.cc:1035] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260806 03:51:25.394966 4276 consensus_queue.cc:1035] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 03:51:25.444029 4428 log.cc:924] Time spent T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.055s user 0.002s sys 0.000s
I20260806 03:51:25.444679 2478 raft_consensus.cc:1261] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:25.499644 2369 raft_consensus.cc:1261] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:25.504832 4346 raft_consensus.cc:348] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:25.520190 4346 raft_consensus.cc:729] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:25.521365 4346 consensus_queue.cc:260] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:25.526180 4366 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap starting.
I20260806 03:51:25.557631 4346 ts_tablet_manager.cc:1428] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.190s user 0.011s sys 0.006s
W20260806 03:51:25.565944 4435 log.cc:924] Time spent T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.071s user 0.001s sys 0.000s
I20260806 03:51:25.483600 4406 consensus_queue.cc:1035] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260806 03:51:25.501645 4236 consensus_queue.cc:1035] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260806 03:51:25.631695 2581 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:25.636987 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260806 03:51:25.653666 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 45071.7us
I20260806 03:51:25.655151 2392 tablet_copy_service.cc:342] P dff94100ffc5485f92cbe6a85addf0b2: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-dfa9a0499fc04e10aefa0602a7adbbb1 received from {username='slave'} at 127.0.0.1:37370
I20260806 03:51:25.675086 2392 tablet_copy_service.cc:433] P dff94100ffc5485f92cbe6a85addf0b2: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-dfa9a0499fc04e10aefa0602a7adbbb1 on tablet dfa9a0499fc04e10aefa0602a7adbbb1 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:25.658010 4385 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
W20260806 03:51:25.712327 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 75882.1us
W20260806 03:51:25.805145 4436 log.cc:924] Time spent T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.175s user 0.000s sys 0.005s
W20260806 03:51:25.832813 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 198970us
W20260806 03:51:25.952512 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 82133.9us
I20260806 03:51:25.958711 2480 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:25.971422 4391 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:25.985410 2232 catalog_manager.cc:5495] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:51:26.018082 4453 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260806 03:51:26.136829 4391 tablet_bootstrap.cc:492] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:26.146091 4391 ts_tablet_manager.cc:1397] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 0.868s user 0.036s sys 0.039s
I20260806 03:51:26.155979 2373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:26.158744 2373 raft_consensus.cc:595] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
W20260806 03:51:26.260563 4462 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
W20260806 03:51:26.284190 4433 log.cc:924] Time spent T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260806 03:51:26.301215 4391 raft_consensus.cc:348] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:26.310763 4391 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:26.313970 4391 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:26.327577 4391 ts_tablet_manager.cc:1428] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.181s user 0.009s sys 0.005s
I20260806 03:51:26.338466 2233 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33762 (request call id 218) took 1185 ms. Trace:
I20260806 03:51:26.355469 2233 rpcz_store.cc:276] 0806 03:51:25.131992 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:25.132242 (+ 250us) service_pool.cc:227] Handling call
0806 03:51:26.317647 (+1185405us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:25.261828 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:25.262482 (+ 654us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:25.262522 (+ 40us) write_op.cc:435] Acquired schema lock
0806 03:51:25.262542 (+ 20us) tablet.cc:660] Decoding operations
0806 03:51:25.262968 (+ 426us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:25.263031 (+ 63us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:25.263069 (+ 38us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:51:25.263312 (+ 243us) tablet.cc:699] Row locks acquired
0806 03:51:25.263337 (+ 25us) write_op.cc:260] PREPARE: finished
0806 03:51:25.263540 (+ 203us) write_op.cc:270] Start()
0806 03:51:25.268311 (+ 4771us) write_op.cc:276] Timestamp: P: 1785988285263474 usec, L: 0
0806 03:51:25.268344 (+ 33us) op_driver.cc:331] REPLICATION: starting
0806 03:51:25.287216 (+ 18872us) log.cc:842] Serialized 535 byte log entry
0806 03:51:25.651359 (+364143us) op_driver.cc:444] REPLICATION: finished
0806 03:51:25.675555 (+ 24196us) write_op.cc:301] APPLY: starting
0806 03:51:25.801997 (+126442us) spinlock_profiling.cc:243] Waited 9.01 ms on lock 0x7b6000070dc8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d8040c361 00007f0d80571a3b 00007f0d80574064 00007f0d804eb3bb 00007f0d803ed343 00007f0d803ef3f8 00007f0d804e9b14 00007f0d804d5dfa 00007f0d804d9dd1 00007f0d804d9d89 00007f0d804d9d19
0806 03:51:25.813479 (+ 11482us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:25.826533 (+ 13054us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:25.834223 (+ 7690us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:25.882259 (+ 48036us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:25.895691 (+ 13432us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:51:25.901025 (+ 5334us) write_op.cc:312] APPLY: finished
0806 03:51:25.921044 (+ 20019us) log.cc:842] Serialized 23 byte log entry
0806 03:51:25.946170 (+ 25126us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:25.952194 (+ 6024us) write_op.cc:454] Released schema lock
0806 03:51:25.952237 (+ 43us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:25.962310 (+ 10073us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:25.962361 (+ 51us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":32185,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":8199,"apply.run_cpu_time_us":6940,"apply.run_wall_time_us":292942,"num_ops":1,"prepare.queue_time_us":106934,"prepare.run_cpu_time_us":3615,"prepare.run_wall_time_us":25825,"replication_time_us":375714,"spinlock_wait_cycles":20714368}]]}
I20260806 03:51:26.374750 2373 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:26.396643 2235 catalog_manager.cc:5495] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 2, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 2 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:26.418839 2373 raft_consensus.cc:595] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
W20260806 03:51:26.451892 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.009s sys 0.015s
W20260806 03:51:26.454735 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.519s user 0.009s sys 0.015s
W20260806 03:51:26.455224 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 519438us
W20260806 03:51:26.513684 2480 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.526s user 0.011s sys 0.026s
W20260806 03:51:26.514287 2480 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.526s user 0.011s sys 0.026s
I20260806 03:51:26.574024 4352 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:26.579669 3591 raft_consensus.cc:1203] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:26.587778 4243 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:26.588547 4243 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:26.592864 2433 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.016s sys 0.032s
W20260806 03:51:26.594192 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.029s sys 0.043s
W20260806 03:51:26.595400 2433 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.516s user 0.017s sys 0.032s
W20260806 03:51:26.595562 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.004s sys 0.024s
W20260806 03:51:26.595498 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.029s sys 0.043s
W20260806 03:51:26.593770 2429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.573s user 0.012s sys 0.017s
W20260806 03:51:26.597548 2429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.012s sys 0.017s
W20260806 03:51:26.597450 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 577931us
W20260806 03:51:26.597059 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.005s sys 0.024s
W20260806 03:51:26.595682 2438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.007s sys 0.019s
W20260806 03:51:26.598745 2438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.714s user 0.008s sys 0.020s
W20260806 03:51:26.598061 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 577037us
W20260806 03:51:26.599363 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 579043us
W20260806 03:51:26.600250 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 386681us
I20260806 03:51:26.598024 4352 tablet_bootstrap.cc:492] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
W20260806 03:51:26.602208 2537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.513s user 0.014s sys 0.016s
I20260806 03:51:26.606101 4236 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:26.632257 4352 ts_tablet_manager.cc:1397] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 1.420s user 0.044s sys 0.047s
W20260806 03:51:26.664114 2578 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.004s sys 0.028s
I20260806 03:51:26.666759 2577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:47302
W20260806 03:51:26.673060 2578 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.685s user 0.005s sys 0.028s
I20260806 03:51:26.673363 2577 raft_consensus.cc:595] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:26.761539 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 142737us
W20260806 03:51:26.761600 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 89593.8us
W20260806 03:51:26.760401 4373 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.787s user 0.020s sys 0.056s
I20260806 03:51:26.664744 4236 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:26.766654 4243 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:26.779711 4307 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:26.794396 4373 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.823s user 0.021s sys 0.057s
I20260806 03:51:26.895689 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:26.982558 2327 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.743s user 0.022s sys 0.019s
W20260806 03:51:26.983213 2327 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.117s user 0.022s sys 0.019s
W20260806 03:51:27.012841 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 117265us
I20260806 03:51:26.990082 4243 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:26.989593 4379 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:27.038261 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 273048us
I20260806 03:51:27.007972 2496 tablet_copy_service.cc:342] P 608e57d72aae496aaf8613622153aa0c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-47ed8dc77b144ba08169a0090b787536 received from {username='slave'} at 127.0.0.1:56244
I20260806 03:51:26.977236 4307 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.044543 4243 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.059580 4236 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:27.060367 4236 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.060672 4243 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:27.060885 4236 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.048008 4379 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.063305 4236 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:27.063661 4379 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:27.049556 2496 tablet_copy_service.cc:433] P 608e57d72aae496aaf8613622153aa0c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-47ed8dc77b144ba08169a0090b787536 on tablet 47ed8dc77b144ba08169a0090b787536 with peer cf6042e85bef47c4aac9d064d91501d1
W20260806 03:51:27.076525 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 176302us
I20260806 03:51:27.066076 4307 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:27.090646 4456 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.098595 4458 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.092877 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:27.091343 4456 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.109072 4456 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:27.085677 4481 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.098737 4475 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.099417 4458 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.112419 4424 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.114493 4444 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:27.131214 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 114471us
I20260806 03:51:27.142549 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:27.137619 4367 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.173177 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:27.174587 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 170813us
I20260806 03:51:27.119869 4481 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.177537 4481 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:27.174782 4487 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.178182 4487 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.180572 4487 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:27.193151 4339 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.194088 4339 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.197981 4449 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.198637 4449 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.202451 4449 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:27.206136 4339 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:27.214385 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 601281us
I20260806 03:51:27.126447 4475 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.223536 4475 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:27.167124 4307 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:27.274698 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 47729.4us
W20260806 03:51:27.281652 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 164101us
I20260806 03:51:27.131342 4424 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.142521 4444 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.226249 4367 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:27.625605 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 349407us
I20260806 03:51:27.631820 4475 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.632514 4475 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.635170 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:27.637009 4367 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:27.617587 2479 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.011s sys 0.011s
W20260806 03:51:27.283922 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 261567us
W20260806 03:51:27.637826 2479 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s user 0.011s sys 0.011s
I20260806 03:51:27.637337 4475 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:27.257367 4339 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:27.506533 4508 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.354876 4243 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:27.460621 4503 log.cc:924] Time spent T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.115s user 0.004s sys 0.000s
W20260806 03:51:27.362129 4500 log.cc:924] Time spent T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.223s user 0.000s sys 0.002s
W20260806 03:51:27.651438 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 494913us
I20260806 03:51:27.655056 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:27.678431 2579 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:27.609912 4474 raft_consensus.cc:979] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower dff94100ffc5485f92cbe6a85addf0b2 to start an election
W20260806 03:51:27.608549 4510 log.cc:924] Time spent T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260806 03:51:27.642716 4164 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.455615 4505 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.624640 4472 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:27.354219 4458 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:27.685710 4164 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:27.423977 4484 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.400s user 0.001s sys 0.002s
W20260806 03:51:27.377650 4499 log.cc:924] Time spent T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
I20260806 03:51:27.422667 4408 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.276476 4307 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.689674 4444 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:27.691488 4335 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.649442 4511 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.680739 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:27.243731 4229 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:27.700969 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 62798us
I20260806 03:51:27.681032 4449 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.739059 4474 raft_consensus.cc:1067] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Signalling peer dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:27.745963 4164 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:27.773926 4491 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.692899 4505 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.722817 4352 raft_consensus.cc:348] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:27.774955 4531 log.cc:924] Time spent T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.076s user 0.000s sys 0.001s
I20260806 03:51:27.775918 4358 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.657472 4528 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.811929 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:27.812232 2478 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:51:27.739979 4335 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.776450 4511 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:27.675547 4339 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.741920 4408 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.683141 4508 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.680555 4243 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:27.722465 4472 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.109330 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 1866) took 2145 ms. Trace:
I20260806 03:51:27.816623 4229 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:28.215967 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.001s sys 0.005s
I20260806 03:51:28.340071 4236 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:28.338543 4454 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:28.340157 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 562428us
I20260806 03:51:28.342155 4488 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:28.343091 4373 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:28.343662 4412 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:28.344261 4234 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:28.344950 4307 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.345512 4497 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:28.346264 4522 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:28.349261 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:28.348944 4358 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.349774 4469 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:28.323583 4352 raft_consensus.cc:729] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:28.351670 4243 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.352481 2584 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:28.353632 4511 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:28.006831 4367 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:28.354746 4536 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:28.355070 4527 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:27.884166 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:28.356307 4481 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:28.355937 4528 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.235975 4379 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:28.110024 4257 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:28.106063 4515 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:27.936988 4404 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:28.357898 4229 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.358677 4385 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:28.359709 4339 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:28.360788 2478 raft_consensus.cc:595] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:28.465277 4379 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.365556 4458 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:27.905694 4491 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.367148 4444 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:28.277189 4164 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:28.174203 4474 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:28.323896 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:28.362061 4526 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:28.368916 4535 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:51:28.338559 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.003s sys 0.005s
I20260806 03:51:28.467285 4527 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.362196 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.170821 2373 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
W20260806 03:51:30.171792 2373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1438) took 1810 ms (client timeout 1716 ms). Trace:
W20260806 03:51:30.172370 2373 rpcz_store.cc:269] 0806 03:51:28.361732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:28.361849 (+ 117us) service_pool.cc:227] Handling call
0806 03:51:30.171645 (+1809796us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:28.362464 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.177277 2377 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:28.360801 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.180274 2379 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
W20260806 03:51:30.181017 2379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1436) took 1820 ms (client timeout 1975 ms). Trace:
W20260806 03:51:30.181411 2379 rpcz_store.cc:269] 0806 03:51:28.360417 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:28.360542 (+ 125us) service_pool.cc:227] Handling call
0806 03:51:30.180931 (+1820389us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:28.477761 2574 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
W20260806 03:51:30.216740 2574 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 822) took 1896 ms (client timeout 1527 ms). Trace:
W20260806 03:51:30.217301 2574 rpcz_store.cc:269] 0806 03:51:28.319809 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:28.348998 (+ 29189us) service_pool.cc:227] Handling call
0806 03:51:30.216111 (+1867113us) spinlock_profiling.cc:243] Waited 70.8 ms on lock 0x7b54001897d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:30.216425 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":162764544}
I20260806 03:51:28.336701 2578 rpcz_store.cc:276] 0806 03:51:25.963431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:25.972823 (+ 9392us) service_pool.cc:227] Handling call
0806 03:51:25.973225 (+ 402us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:25.983516 (+ 10291us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:25.983565 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:25.988225 (+ 4660us) log.cc:842] Serialized 26 byte log entry
0806 03:51:26.673527 (+685302us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:26.673567 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:26.673630 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:27.423206 (+749576us) spinlock_profiling.cc:243] Waited 5.87 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:28.054168 (+630962us) raft_consensus.cc:1671] finished
0806 03:51:28.054207 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:28.109196 (+ 54989us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12651,"raft.queue_time_us":20232,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":13801,"spinlock_wait_cycles":13506816,"thread_start_us":684979,"threads_started":1,"wal-append.queue_time_us":802448}
I20260806 03:51:28.465566 4536 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.517269 4535 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:28.365620 4538 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:29.454475 4537 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:29.740250 4550 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:29.733700 4554 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.239514 4559 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:51:30.032459 2327 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.680s user 0.041s sys 0.053s
W20260806 03:51:30.040750 2433 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.646s user 0.000s sys 0.016s
I20260806 03:51:30.073366 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1478) took 2150 ms. Trace:
I20260806 03:51:28.350417 4276 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:28.469002 4352 consensus_queue.cc:260] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:28.007074 4374 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:30.257740 4374 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.405040 4375 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.261560 4375 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:28.520807 4408 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.029444 4472 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.455726 4511 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.269500 4374 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.269402 4511 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.495324 4164 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.404479 4234 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.275524 4164 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:30.276485 4511 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.276611 4234 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.277498 2478 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:30.277393 4234 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.278404 4234 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.385869 4236 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.280591 4489 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.280786 4563 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.281564 4489 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.281714 4563 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.467423 4257 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.286757 4257 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:29.012543 4335 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:28.561333 4358 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:29.014505 4366 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 03:51:28.454911 4367 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.561298 4379 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:28.492528 4385 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.390017 4373 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:28.476538 4404 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.299942 4385 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.430446 4412 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.371124 4424 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:28.487787 4444 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.835839 4449 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.314177 4373 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.314499 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 824) took 1977 ms. Trace:
I20260806 03:51:30.315093 4366 tablet_bootstrap.cc:492] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Bootstrap complete.
I20260806 03:51:30.314961 2576 rpcz_store.cc:276] 0806 03:51:28.337046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:28.347079 (+ 10033us) service_pool.cc:227] Handling call
0806 03:51:28.347490 (+ 411us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:30.304505 (+1957015us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:30.304556 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:30.304606 (+ 50us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:30.304637 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:30.313937 (+ 9300us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b5400156654. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:30.314002 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:30.314242 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4484939136}
I20260806 03:51:30.315791 4449 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.431268 4454 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.477603 4458 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.466017 4469 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.320531 4454 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.491535 4474 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:28.458179 4475 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:28.428963 4477 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.325762 4475 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.326336 4474 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:28.466001 4481 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:29.013268 4508 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:28.359655 4507 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.320958 4489 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.333632 4563 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.319154 4505 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:28.427508 4501 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:28.408840 4497 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:28.522203 4491 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:28.425066 4488 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:27.937022 4513 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:28.481199 4515 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:28.443662 4522 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.345757 4515 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:28.518532 4526 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:28.362967 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.088722 4545 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.353938 4522 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:30.353963 2370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1440) took 1991 ms (client timeout 2035 ms). Trace:
I20260806 03:51:29.011622 4528 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:28.456960 4529 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:28.314440 4530 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.335s user 0.001s sys 0.000s
I20260806 03:51:30.092614 4527 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.089309 4542 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.361606 4542 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.364432 4542 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:30.098145 2332 connection.cc:232] RPC call timeout handler was delayed by -2.0577618260867894s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
I20260806 03:51:30.099417 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 834) took 3488 ms. Trace:
I20260806 03:51:30.099512 4471 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:30.365372 2332 connection.cc:232] RPC call timeout handler was delayed by -1.4472003160044551s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1755s for a response
I20260806 03:51:30.366384 4385 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:30.366050 4471 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:30.367123 4385 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.111316 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1469) took 2301 ms. Trace:
I20260806 03:51:30.368188 2480 rpcz_store.cc:276] 0806 03:51:27.809668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:27.812850 (+ 3182us) service_pool.cc:227] Handling call
0806 03:51:30.109604 (+2296754us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:30.110177 (+ 573us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:30.110694 (+ 517us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:30.110722 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:30.110797 (+ 75us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:30.110828 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:30.110905 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:30.111040 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42448256}
I20260806 03:51:30.369436 4385 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.369983 4385 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.370182 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.370743 4527 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.370947 2480 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
W20260806 03:51:30.112632 3545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.791s user 0.005s sys 0.015s
I20260806 03:51:30.114554 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:30.118422 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.664s user 0.022s sys 0.008s
W20260806 03:51:30.372241 3545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.051s user 0.006s sys 0.015s
W20260806 03:51:30.372565 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.918s user 0.022s sys 0.008s
W20260806 03:51:30.372792 2482 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1477) took 2515 ms (client timeout 1716 ms). Trace:
W20260806 03:51:30.373232 2482 rpcz_store.cc:269] 0806 03:51:27.857729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:27.857916 (+ 187us) service_pool.cc:227] Handling call
0806 03:51:30.114045 (+2256129us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:51:30.372694 (+258649us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":12864384}
I20260806 03:51:30.374648 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:30.129628 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.713s user 0.004s sys 0.011s
W20260806 03:51:30.376255 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.966s user 0.005s sys 0.011s
I20260806 03:51:30.376389 4505 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:30.376652 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 2.05321e+06us
I20260806 03:51:30.377198 4505 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.380122 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.381054 3592 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:30.381902 4515 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.381726 4385 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.382788 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.382651 4515 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.383858 4508 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:30.384063 4505 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.384203 4385 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.384719 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.384510 4508 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.385452 2482 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:30.387084 4508 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.387465 4528 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:30.388304 4528 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.390401 4528 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.390812 3593 raft_consensus.cc:1203] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:30.391201 4528 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:30.133198 3542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.497s user 0.007s sys 0.006s
W20260806 03:51:30.392357 3542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.757s user 0.008s sys 0.007s
W20260806 03:51:30.392724 2538 connection.cc:232] RPC call timeout handler was delayed by -1.7404802890960127s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1655s for a response
W20260806 03:51:30.393246 2538 connection.cc:232] RPC call timeout handler was delayed by -1.6051720639225096s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
W20260806 03:51:30.393536 2538 connection.cc:232] RPC call timeout handler was delayed by -1.5487802890129387s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18680000000000002s for a response
W20260806 03:51:30.393843 2538 connection.cc:232] RPC call timeout handler was delayed by -1.4155849209055305s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1975s for a response
I20260806 03:51:30.398639 2538 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:30.399777 4491 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:30.399847 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.400728 2373 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:30.133213 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 45) took 3241 ms. Trace:
W20260806 03:51:30.135144 3543 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.472s user 0.009s sys 0.006s
I20260806 03:51:30.402187 2374 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:40728
I20260806 03:51:30.402932 2374 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:30.403223 2374 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:30.144184 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:30.185376 2537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.794s user 0.011s sys 0.010s
W20260806 03:51:30.190877 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.857s user 0.019s sys 0.001s
W20260806 03:51:30.207211 2584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 820) took 2552 ms (client timeout 1996 ms). Trace:
W20260806 03:51:30.406934 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.074s user 0.019s sys 0.001s
W20260806 03:51:30.224689 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.814s user 0.016s sys 0.073s
I20260806 03:51:30.226245 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 816) took 2583 ms. Trace:
W20260806 03:51:30.407069 2537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.016s user 0.012s sys 0.011s
I20260806 03:51:30.231249 4537 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.236285 4536 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.236470 4539 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.242167 2484 rpcz_store.cc:276] 0806 03:51:27.922843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:28.318962 (+396119us) service_pool.cc:227] Handling call
0806 03:51:30.071289 (+1752327us) spinlock_profiling.cc:243] Waited 16.9 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:30.071891 (+ 602us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:30.072518 (+ 627us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:30.072575 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:30.072644 (+ 69us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:30.072679 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:30.072797 (+ 118us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:30.073054 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":38959872}
W20260806 03:51:30.242686 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 1.74581e+06us
W20260806 03:51:30.244257 2433 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.849s user 0.000s sys 0.017s
I20260806 03:51:30.240121 4554 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.243772 4543 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:30.244683 2327 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.892s user 0.042s sys 0.053s
I20260806 03:51:30.241026 4559 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.238415 4550 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.229326 4538 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.248533 4276 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.263890 4375 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.272660 4352 ts_tablet_manager.cc:1428] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 3.601s user 0.028s sys 0.055s
I20260806 03:51:30.303043 4404 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.303828 4367 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.304404 4236 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:30.312322 3547 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.729s (SENT)
I20260806 03:51:30.314090 4444 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.316710 4366 ts_tablet_manager.cc:1397] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Time spent bootstrapping tablet: real 4.791s user 0.064s sys 0.098s
I20260806 03:51:28.308106 4456 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.417266 4543 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.456899 4456 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.318953 4234 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.323369 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.321085 4458 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.326282 4477 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.322856 4469 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.318996 4412 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.330435 4481 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.484916 4554 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.331501 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.333693 4507 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:30.337347 4501 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.340764 4475 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.341686 4497 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.343554 4488 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.349658 4513 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.352434 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:30.354606 2370 rpcz_store.cc:269] 0806 03:51:28.362470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:28.362632 (+ 162us) service_pool.cc:227] Handling call
0806 03:51:30.353780 (+1991148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:30.354561 4545 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:30.356523 2436 connection.cc:232] RPC call timeout handler was delayed by -1.3249543751589954s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1527s for a response
I20260806 03:51:30.356475 4526 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.357836 4529 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.371513 4527 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.391889 4528 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:30.402469 3543 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.739s user 0.009s sys 0.006s
W20260806 03:51:30.184008 2377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1439) took 1822 ms (client timeout 2243 ms). Trace:
W20260806 03:51:30.499365 2377 rpcz_store.cc:269] 0806 03:51:28.362080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:28.362209 (+ 129us) service_pool.cc:227] Handling call
0806 03:51:30.178094 (+1815885us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:30.158581 4509 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.231606 4535 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:30.407003 2584 rpcz_store.cc:269] 0806 03:51:27.654889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:27.655024 (+ 135us) service_pool.cc:227] Handling call
0806 03:51:30.206492 (+2551468us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b540039bcd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:30.206836 (+ 344us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1368489216}
I20260806 03:51:30.408551 4257 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.407949 2575 rpcz_store.cc:276] 0806 03:51:27.642505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:27.644371 (+ 1866us) service_pool.cc:227] Handling call
0806 03:51:28.319799 (+675428us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:28.482512 (+162713us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:28.482557 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:28.482617 (+ 60us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:28.482660 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:30.225558 (+1742898us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b540013b0d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:30.225639 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:30.225973 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":366477568}
W20260806 03:51:30.409340 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.999s user 0.017s sys 0.073s
W20260806 03:51:30.407465 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 2.75665e+06us
W20260806 03:51:30.366061 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0840691851917654s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19990000000000002s for a response
I20260806 03:51:30.365175 2369 rpcz_store.cc:276] 0806 03:51:26.610317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:26.610587 (+ 270us) service_pool.cc:227] Handling call
0806 03:51:26.610917 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:26.611263 (+ 346us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:26.611305 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:26.625706 (+ 14401us) log.cc:842] Serialized 26 byte log entry
0806 03:51:26.790766 (+165060us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:26.790810 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:26.790886 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:30.098721 (+3307835us) raft_consensus.cc:1671] finished
0806 03:51:30.098770 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:30.099111 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7505,"raft.queue_time_us":1691,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":44535,"thread_start_us":114343,"threads_started":1,"wal-append.queue_time_us":402604,"wal-append.run_cpu_time_us":40372,"wal-append.run_wall_time_us":2446989}
I20260806 03:51:30.488557 4542 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:30.502633 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0618300109636039s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1855s for a response
I20260806 03:51:30.414400 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:30.407584 2474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1476) took 2595 ms (client timeout 1975 ms). Trace:
W20260806 03:51:30.419405 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 2.02309e+06us
W20260806 03:51:30.431500 4352 ts_tablet_manager.cc:1460] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Tablet startup took 3763ms
I20260806 03:51:30.440832 4321 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:30.441511 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1471) took 2631 ms. Trace:
I20260806 03:51:30.415769 4537 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.401623 2479 rpcz_store.cc:276] 0806 03:51:26.892133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:26.893194 (+ 1061us) service_pool.cc:227] Handling call
0806 03:51:26.990068 (+ 96874us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:27.065111 (+ 75043us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:27.065152 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:27.065657 (+ 505us) log.cc:842] Serialized 26 byte log entry
0806 03:51:27.638235 (+572578us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:51:27.807298 (+169063us) log.cc:842] Serialized 14 byte log entry
0806 03:51:28.341315 (+534017us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:30.131974 (+1790659us) spinlock_profiling.cc:243] Waited 74.6 ms on lock 0x7b5400350cd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:30.132150 (+ 176us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:30.132307 (+ 157us) raft_consensus.cc:1671] finished
0806 03:51:30.132534 (+ 227us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:30.132942 (+ 408us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":528321,"prepare.queue_time_us":715782,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":539661,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":6940,"spinlock_wait_cycles":171498240,"thread_start_us":719335,"threads_started":2,"wal-append.queue_time_us":558710,"wal-append.run_cpu_time_us":15186,"wal-append.run_wall_time_us":1889341}
I20260806 03:51:30.483881 4444 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:30.503837 2474 rpcz_store.cc:269] 0806 03:51:27.811932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:27.884166 (+ 72234us) service_pool.cc:227] Handling call
0806 03:51:30.143715 (+2259549us) spinlock_profiling.cc:243] Waited 18 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65bce 000000000034aea9 00007f0d81991a30 00007f0d81a577b4 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:51:30.407478 (+263763us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41379072}
I20260806 03:51:30.506619 4559 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.506321 4444 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.474833 4471 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.474795 4456 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.509418 4257 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.510978 2537 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:30.512486 4367 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:30.513067 4545 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.413405 4539 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:30.494822 2436 connection.cc:232] RPC call timeout handler was delayed by -1.1503543751314282s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17210000000000003s for a response
W20260806 03:51:30.495468 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 183361us
I20260806 03:51:30.496760 4550 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.486074 4385 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:30.505218 2483 rpcz_store.cc:276] 0806 03:51:27.810380 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:27.815928 (+ 5548us) service_pool.cc:227] Handling call
0806 03:51:30.065641 (+2249713us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:30.066415 (+ 774us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:30.426572 (+360157us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:30.426619 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:30.426683 (+ 64us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:30.426764 (+ 81us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:30.439582 (+ 12818us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b54000af5d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:30.439653 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:30.441271 (+ 1618us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":869155456}
I20260806 03:51:30.501098 4458 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:30.492609 4276 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.497375 4472 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.528901 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.529152 4472 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.529574 4458 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.530090 2436 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:30.531642 4472 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.507460 2374 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.532925 4559 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:30.533288 4527 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.533361 4559 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.509479 2480 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:30.505105 4321 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.536588 2374 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.528863 4539 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.542402 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.542544 2372 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:30.543226 4529 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.546207 4528 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:30.505149 4352 ts_tablet_manager.cc:1462] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Trace:
0806 03:51:23.029821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:23.030228 (+ 407us) service_pool.cc:227] Handling call
0806 03:51:23.271636 (+241408us) ts_tablet_manager.cc:928] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
0806 03:51:24.221786 (+950150us) tablet_copy_client.cc:361] Tablet copy session begun
0806 03:51:24.469365 (+247579us) tablet_metadata.cc:641] Metadata flushed
0806 03:51:24.505695 (+ 36330us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 03:51:24.835509 (+329814us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 03:51:24.836002 (+ 493us) tablet_replica.cc:205] Creating consensus instance
0806 03:51:24.883506 (+ 47504us) ts_tablet_manager.cc:983] Replica registered.
0806 03:51:24.883589 (+ 83us) inbound_call.cc:173] Queueing success response
0806 03:51:25.212455 (+328866us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0806 03:51:26.677684 (+1465229us) ts_tablet_manager.cc:1429] Starting tablet replica
0806 03:51:27.099282 (+421598us) tablet_replica.cc:259] Starting instrumentation
0806 03:51:27.099982 (+ 700us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":440,"raft.queue_time_us":648,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":303,"tablet-copy.queue_time_us":238871,"tablet-download-pool.queue_time_us":417,"tablet-download-pool.run_cpu_time_us":2033,"tablet-download-pool.run_wall_time_us":161925,"thread_start_us":301954,"threads_started":2}
I20260806 03:51:30.531965 4458 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.548344 4507 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.550078 4513 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.549129 2537 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:30.551498 4321 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.555642 4385 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.534664 4559 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:30.501384 4509 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:30.559188 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 48263us
I20260806 03:51:30.570300 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:30.634479 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 59746.1us
I20260806 03:51:30.604367 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.590636 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.647324 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.556093 4538 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.553541 4550 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.734378 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.728466 4385 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.739120 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.745461 4550 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:30.575268 4530 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.056s user 0.002s sys 0.000s
I20260806 03:51:30.746958 3591 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:30.748703 4497 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:30.753218 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.753576 4497 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.748942 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.554283 4444 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.589259 4501 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.567922 4515 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.557121 4542 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.753340 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.753914 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.589227 4375 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:30.749955 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.754137 2375 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:30.633281 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.600939 4257 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.756964 2372 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
W20260806 03:51:30.763005 2534 connection.cc:232] RPC call timeout handler was delayed by -1.1527393769938499s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1716s for a response
I20260806 03:51:30.755293 2370 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
W20260806 03:51:30.764765 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 173623us
I20260806 03:51:30.575055 4543 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.749982 4550 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.763994 4375 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:30.762800 4276 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.796224 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.795277 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.816447 2377 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
W20260806 03:51:30.791597 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 69856.8us
I20260806 03:51:30.761560 4456 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:30.810913 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.813032 4375 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:30.786389 2369 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:30.755323 4477 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:30.791561 2374 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:30.827860 4509 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.813992 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.828706 2534 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:30.829401 2375 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:30.830331 2534 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:30.831168 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.832055 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.836199 2534 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:30.839918 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.808398 4513 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:30.843143 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.769626 4501 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:30.845466 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.846827 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.847193 4529 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:30.847961 4366 raft_consensus.cc:348] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:30.849210 4367 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:30.849238 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.857007 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.862284 4513 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.864494 3593 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:30.865278 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.868852 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.873461 4491 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:30.874035 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.874068 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.875152 4542 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:30.857682 4529 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:30.876082 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.876325 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.876771 4497 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:30.878886 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.879704 2479 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:30.877641 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.889813 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.890046 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.881817 4367 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:30.853201 4276 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.891742 4501 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:30.898018 3590 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:30.920228 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.920414 3591 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:30.898701 3594 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
W20260806 03:51:30.924413 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 175072us
I20260806 03:51:30.882071 4366 raft_consensus.cc:729] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Initialized, Role: LEARNER
I20260806 03:51:30.919750 4529 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:30.923070 3588 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:30.917618 4491 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:30.948024 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:30.949015 3593 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:30.927647 4367 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:30.881762 4456 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:30.965169 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.958870 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.929309 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.973066 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.974529 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.963874 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.976312 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.976859 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:30.977094 4366 consensus_queue.cc:260] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:30.979645 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:30.992187 4501 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:31.007586 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 174794us
I20260806 03:51:30.952891 4491 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:31.012959 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 76924.9us
I20260806 03:51:31.008486 4542 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.027460 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.037395 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.041675 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.045109 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.045379 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.045936 2477 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:31.046866 4568 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.046983 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.047616 4568 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.048442 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.049134 2482 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:31.048863 2534 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:31.049989 4568 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:31.050972 4406 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:31.051229 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.051416 4406 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.051802 4406 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:31.052815 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.045205 4472 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.034618 4501 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:31.060261 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.060410 4472 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.060523 4501 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.062717 4472 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:31.063256 4472 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.047077 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.072616 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.074290 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.063791 4472 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.085001 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.083868 2474 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:31.088836 2479 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:31.063489 4501 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:31.096136 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.097460 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.097863 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.099478 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:31.090063 3620 debug-util.cc:398] Leaking SignalData structure 0x7b08002ea7a0 after lost signal to thread 735
I20260806 03:51:31.086416 4542 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:31.101162 3620 debug-util.cc:398] Leaking SignalData structure 0x7b080025d8c0 after lost signal to thread 2283
I20260806 03:51:31.101333 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.101346 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.102053 4542 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:31.102111 2379 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:31.102228 3587 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
W20260806 03:51:31.102674 3620 debug-util.cc:398] Leaking SignalData structure 0x7b0800357da0 after lost signal to thread 2413
W20260806 03:51:31.103003 3620 debug-util.cc:398] Leaking SignalData structure 0x7b0800357dc0 after lost signal to thread 2414
I20260806 03:51:31.103799 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.113443 4513 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:31.118577 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.119673 2235 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38220 (request call id 118) took 4734 ms. Trace:
I20260806 03:51:31.118918 4472 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:31.120162 3620 debug-util.cc:398] Leaking SignalData structure 0x7b08003505a0 after lost signal to thread 2516
I20260806 03:51:31.129850 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.133292 2233 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33762 (request call id 219) took 4655 ms. Trace:
I20260806 03:51:31.132941 4576 consensus_queue.cc:1046] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:31.133323 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.129814 4366 ts_tablet_manager.cc:1428] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Time spent starting tablet: real 0.692s user 0.015s sys 0.011s
I20260806 03:51:31.148375 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.140471 2575 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.133729 2233 rpcz_store.cc:276] 0806 03:51:26.477629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:26.477968 (+ 339us) service_pool.cc:227] Handling call
0806 03:51:31.133058 (+4655090us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31552,"reported_tablets":5}
I20260806 03:51:31.128481 4456 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:31.129855 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.155236 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.151351 2534 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:31.157038 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:31.150540 4366 ts_tablet_manager.cc:1460] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Tablet startup took 712ms
I20260806 03:51:31.160377 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.153070 4559 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.162299 2534 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:31.158450 3620 debug-util.cc:398] Leaking SignalData structure 0x7b08002deb20 after lost signal to thread 2620
I20260806 03:51:31.153082 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.168953 2232 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33786 (request call id 209) took 5583 ms. Trace:
I20260806 03:51:31.170389 4559 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:31.155252 4477 raft_consensus.cc:3032] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:31.171114 4477 raft_consensus.cc:729] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:31.173650 4456 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.174247 4456 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.174738 4456 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.168191 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.178458 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.181820 4576 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.182303 4576 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.182092 4477 consensus_queue.cc:260] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.183398 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.184145 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.184638 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.185052 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.185393 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.182729 4576 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.185751 2581 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.161108 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.189867 3547 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:31.190626 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:31.167271 3620 debug-util.cc:398] Leaking SignalData structure 0x7b0800271ea0 after lost signal to thread 2622
I20260806 03:51:31.163914 4276 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:31.120096 2235 rpcz_store.cc:276] 0806 03:51:26.385340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:26.385633 (+ 293us) service_pool.cc:227] Handling call
0806 03:51:31.119449 (+4733816us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:28.345744 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:28.346381 (+ 637us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:28.346426 (+ 45us) write_op.cc:435] Acquired schema lock
0806 03:51:28.346447 (+ 21us) tablet.cc:660] Decoding operations
0806 03:51:28.346812 (+ 365us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:28.346866 (+ 54us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:28.346898 (+ 32us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:51:28.347101 (+ 203us) tablet.cc:699] Row locks acquired
0806 03:51:28.347122 (+ 21us) write_op.cc:260] PREPARE: finished
0806 03:51:28.347257 (+ 135us) write_op.cc:270] Start()
0806 03:51:28.347389 (+ 132us) write_op.cc:276] Timestamp: P: 1785988288347211 usec, L: 0
0806 03:51:28.347410 (+ 21us) op_driver.cc:331] REPLICATION: starting
0806 03:51:28.384062 (+ 36652us) log.cc:842] Serialized 530 byte log entry
0806 03:51:30.849860 (+2465798us) op_driver.cc:444] REPLICATION: finished
0806 03:51:30.869869 (+ 20009us) write_op.cc:301] APPLY: starting
0806 03:51:30.869966 (+ 97us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:30.870106 (+ 140us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:30.870280 (+ 174us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:30.870588 (+ 308us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:30.870763 (+ 175us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:51:30.870808 (+ 45us) write_op.cc:312] APPLY: finished
0806 03:51:30.871023 (+ 215us) log.cc:842] Serialized 23 byte log entry
0806 03:51:30.871200 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:30.871322 (+ 122us) write_op.cc:454] Released schema lock
0806 03:51:30.871360 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:30.871581 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:30.871604 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":123378,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":19711,"apply.run_cpu_time_us":1942,"apply.run_wall_time_us":1942,"num_ops":1,"prepare.queue_time_us":1935591,"prepare.run_cpu_time_us":3237,"prepare.run_wall_time_us":61492,"replication_time_us":2471370,"thread_start_us":78917,"threads_started":1}]]}
I20260806 03:51:31.194579 2534 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:31.199390 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.198992 4491 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:31.200196 2577 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.201159 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.202359 2534 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:31.186728 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.195353 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.202415 4491 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:31.195569 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.204663 4474 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.204887 4501 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:31.205148 4474 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.205411 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.205818 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.205654 4474 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.206633 2581 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:31.193050 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.207898 2584 raft_consensus.cc:1226] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:31.208372 2534 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:31.212057 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 61449.3us
I20260806 03:51:31.207442 2576 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.185408 4477 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.219059 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.219501 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.229648 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.229641 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.232055 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.232167 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.219278 4257 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.233690 4257 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.234140 4257 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.236796 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.205432 4501 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:31.231083 2572 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.232239 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.241369 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.242195 2374 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:31.239647 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:31.246654 2575 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.248430 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.249084 2583 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.238533 4406 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.243068 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.252730 2579 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.257853 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.258556 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.245637 2436 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:31.261092 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.169533 2232 rpcz_store.cc:276] 0806 03:51:25.510352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:25.510514 (+ 162us) service_pool.cc:227] Handling call
0806 03:51:25.638138 (+127624us) spinlock_profiling.cc:243] Waited 95.9 ms on lock 0x7b54000e3154. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4aaf0 00007f0d7cacdf2a 00007f0d82a1fd6c 00007f0d82b19c3e 00007f0d80e2bfb4 00007f0d80e2bf42 00007f0d80e2be71 00007f0d80e2bdec 00007f0d80e2b0a2 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:31.089326 (+5451188us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:27.101676 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:27.141781 (+ 40105us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:27.147182 (+ 5401us) write_op.cc:435] Acquired schema lock
0806 03:51:27.147203 (+ 21us) tablet.cc:660] Decoding operations
0806 03:51:27.222152 (+ 74949us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:27.222225 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:27.222257 (+ 32us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:51:27.222508 (+ 251us) tablet.cc:699] Row locks acquired
0806 03:51:27.222526 (+ 18us) write_op.cc:260] PREPARE: finished
0806 03:51:27.222715 (+ 189us) write_op.cc:270] Start()
0806 03:51:27.222864 (+ 149us) write_op.cc:276] Timestamp: P: 1785988287222662 usec, L: 0
0806 03:51:27.222887 (+ 23us) op_driver.cc:331] REPLICATION: starting
0806 03:51:27.223740 (+ 853us) log.cc:842] Serialized 535 byte log entry
0806 03:51:30.537497 (+3313757us) op_driver.cc:444] REPLICATION: finished
0806 03:51:30.802229 (+264732us) write_op.cc:301] APPLY: starting
0806 03:51:30.802392 (+ 163us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:30.802550 (+ 158us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:30.802729 (+ 179us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:30.818240 (+ 15511us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:30.850703 (+ 32463us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:51:30.868084 (+ 17381us) write_op.cc:312] APPLY: finished
0806 03:51:30.868559 (+ 475us) log.cc:842] Serialized 23 byte log entry
0806 03:51:30.868822 (+ 263us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:30.868937 (+ 115us) write_op.cc:454] Released schema lock
0806 03:51:30.869006 (+ 69us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:30.869346 (+ 340us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:30.869379 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":599416,"reported_tablets":4,"spinlock_wait_cycles":220509568,"child_traces":[["op",{"apply.queue_time_us":220134,"apply.run_cpu_time_us":3262,"apply.run_wall_time_us":77225,"num_ops":1,"prepare.queue_time_us":761370,"prepare.run_cpu_time_us":20288,"prepare.run_wall_time_us":1258116,"raft.queue_time_us":140858,"raft.run_cpu_time_us":22302,"raft.run_wall_time_us":344493,"replication_time_us":3279372,"thread_start_us":953493,"threads_started":4,"wal-append.queue_time_us":405220}]]}
I20260806 03:51:31.264300 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.263959 2379 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:31.263942 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.266585 2436 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:31.243434 4501 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:31.262156 3620 debug-util.cc:398] Leaking SignalData structure 0x7b0800367a80 after lost signal to thread 3624
I20260806 03:51:31.242966 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:31.278309 2534 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:31.220925 4529 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.281114 2436 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:31.283686 4412 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.264240 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.245199 4491 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:31.286732 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.289111 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.289348 2436 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:31.290027 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:31.273101 4580 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.275249 4578 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:31.297598 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.297025 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.296496 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.286573 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.300643 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.303164 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.301450 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.284190 4412 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.305375 2436 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:31.306816 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.326680 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.312704 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.314070 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.314280 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.317195 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.305061 4513 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.319921 4529 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:31.320909 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.288661 4491 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.290341 4559 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.302840 4578 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.297498 4475 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:31.334378 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.358410 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.292200 4580 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.332165 2368 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:31.338829 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.330265 4412 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.360702 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.328552 2374 raft_consensus.cc:2379] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 2: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:31.309702 2376 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:31.343492 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.338342 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.353449 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.343641 4491 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:31.332834 4543 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.338578 4513 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.345263 4559 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.375139 2579 raft_consensus.cc:1226] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:31.375988 2538 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:31.355567 4406 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:31.376729 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:31.356160 4578 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:31.358062 4475 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.379189 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.364246 4580 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.373656 4543 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:31.378584 4513 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.380072 4559 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.394029 3585 raft_consensus.cc:1203] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20260806 03:51:31.394961 4509 raft_consensus.cc:1050] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:31.414485 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:31.161217 4366 ts_tablet_manager.cc:1462] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Trace:
0806 03:51:23.292111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:23.292308 (+ 197us) service_pool.cc:227] Handling call
0806 03:51:23.423484 (+131176us) ts_tablet_manager.cc:928] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Initiating tablet copy from peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
0806 03:51:24.356974 (+933490us) tablet_copy_client.cc:361] Tablet copy session begun
0806 03:51:24.752745 (+395771us) tablet_metadata.cc:641] Metadata flushed
0806 03:51:24.753097 (+ 352us) tablet_copy_client.cc:490] Wrote new tablet metadata
0806 03:51:25.124687 (+371590us) tablet_copy_client.cc:411] Wrote new consensus metadata
0806 03:51:25.168251 (+ 43564us) tablet_replica.cc:205] Creating consensus instance
0806 03:51:25.169075 (+ 824us) ts_tablet_manager.cc:983] Replica registered.
0806 03:51:25.169235 (+ 160us) inbound_call.cc:173] Queueing success response
0806 03:51:25.525744 (+356509us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0806 03:51:30.437479 (+4911735us) ts_tablet_manager.cc:1429] Starting tablet replica
0806 03:51:30.593954 (+156475us) tablet_replica.cc:259] Starting instrumentation
0806 03:51:30.754351 (+160397us) tablet_replica.cc:278] Starting consensus
Metrics: {"dns_us":367,"mutex_wait_us":10119,"raft.queue_time_us":7425,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":193,"tablet-copy.queue_time_us":124633,"tablet-download-pool.queue_time_us":6351,"tablet-download-pool.run_cpu_time_us":12328,"tablet-download-pool.run_wall_time_us":38125,"thread_start_us":159102,"threads_started":2}
I20260806 03:51:31.404136 2474 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:51:31.405750 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.386736 4475 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:31.423504 3594 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
W20260806 03:51:31.384218 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 63840us
I20260806 03:51:31.387818 4529 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:31.420320 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.417181 4576 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.432942 4576 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:31.433698 4375 consensus_meta.cc:287] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.577s user 0.011s sys 0.003s
I20260806 03:51:31.434487 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.396723 4543 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.381008 4367 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.434252 4375 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:31.415952 4509 consensus_queue.cc:237] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:31.449842 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.448554 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.451272 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.457389 2579 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.430630 2474 raft_consensus.cc:595] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:31.458997 2583 raft_consensus.cc:1226] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:31.461236 4375 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:31.461468 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.461730 2581 raft_consensus.cc:1226] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:31.464164 2474 raft_consensus.cc:603] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 03:51:31.501513 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 74231.2us
I20260806 03:51:31.504153 2534 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:31.503337 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:31.508670 4542 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.509092 4542 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.509482 4542 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.511905 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.516705 2534 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:31.517688 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 109163us
I20260806 03:51:31.521195 2538 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:31.521405 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 03:51:31.525269 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.525123 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.527684 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:31.528398 2377 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.453765 4474 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.519227 4375 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:31.516260 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.534176 4528 raft_consensus.cc:2781] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:31.541798 4528 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.542264 4528 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:31.551047 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.559577 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.561100 2579 raft_consensus.cc:2396] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:31.565605 2538 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:31.545678 4529 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:31.570869 4367 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:31.573990 4474 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:31.537277 4375 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.578202 4375 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:31.578970 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:31.579694 2572 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.579845 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.580646 2581 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:31.584820 2578 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:31.590870 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:31.598773 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 66277.8us
I20260806 03:51:31.604741 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.605165 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.593722 4404 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:31.622609 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.623999 2579 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:31.650769 3542 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:31.667560 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:51:31.649339 4474 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:31.596347 4591 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260806 03:51:31.685421 4584 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.686065 4584 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:31.686456 4584 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:31.696414 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 82385.6us
I20260806 03:51:31.702044 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.709211 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.708972 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:31.716487 2578 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
W20260806 03:51:31.734858 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 214054us
I20260806 03:51:31.744330 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:31.695930 4474 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.765012 3587 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:31.764710 4513 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:31.712742 3542 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:31.767237 4513 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:31.767289 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 62737.5us
I20260806 03:51:31.767755 4583 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:31.768327 3542 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:31.773847 4592 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.098s user 0.001s sys 0.000s
I20260806 03:51:31.774121 3542 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:31.775079 4550 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:31.775715 4550 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.776083 4550 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.776430 4550 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.711054 4412 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:31.785255 4257 consensus_meta.cc:287] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.551s user 0.012s sys 0.012s
W20260806 03:51:31.788412 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 242171us
I20260806 03:51:31.788508 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:31.790220 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.785777 4412 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:31.792528 2578 raft_consensus.cc:2379] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:31.795390 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:31.785707 4257 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:31.771646 4591 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.796212 3585 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:31.804272 3594 raft_consensus.cc:2379] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 2: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:31.806488 2538 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:31.799860 2572 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:31.812716 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:31.797892 4257 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:31.822858 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 114624us
I20260806 03:51:31.828006 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.828685 2538 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:31.816695 4591 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:31.780133 4501 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:31.847272 2538 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:31.822259 4568 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260806 03:51:31.857054 4550 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:31.859253 4578 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.862764 4580 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:31.866263 4501 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:31.868678 4550 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:31.869138 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.869586 2597 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-78ff87b2196e4600aed5d8442f9b9bdd received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:31.869957 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.922056 2579 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:31.870376 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.870736 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.923804 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.871116 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:31.927835 2584 raft_consensus.cc:1226] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:31.855444 4591 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:31.907178 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 59145.6us
I20260806 03:51:31.844844 4406 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:31.869992 4404 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:31.920507 2577 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:31.920642 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.871405 2538 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:31.931422 4456 consensus_meta.cc:287] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.746s user 0.012s sys 0.016s
I20260806 03:51:31.933112 3584 raft_consensus.cc:1203] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260806 03:51:31.937054 4580 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:31.938493 3591 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:31.938719 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 93765.6us
I20260806 03:51:31.839299 4603 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:31.941926 4474 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:31.942253 4578 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:31.942842 3592 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:31.945060 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:31.972731 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:31.950449 4613 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
W20260806 03:51:31.978369 4543 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.519s user 0.012s sys 0.004s
I20260806 03:51:31.978525 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.979055 4543 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:31.953722 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 82411us
I20260806 03:51:31.955003 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.955854 3545 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:32.003515 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 47753.5us
I20260806 03:51:32.003643 4550 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:31.957890 4367 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:31.955724 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:31.956528 4456 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.960237 4375 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:31.963351 4491 consensus_meta.cc:287] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.583s user 0.011s sys 0.000s
I20260806 03:51:31.963045 4584 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:31.965755 4475 consensus_meta.cc:287] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.540s user 0.015s sys 0.000s
I20260806 03:51:32.019768 4475 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.020114 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:32.020495 4543 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:32.021040 4584 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:32.021947 4475 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:32.022619 2575 raft_consensus.cc:1226] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:32.023065 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:31.976485 4542 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.024253 2370 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:31.979817 4559 consensus_meta.cc:287] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.558s user 0.012s sys 0.008s
I20260806 03:51:32.025287 4542 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:32.025182 4559 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.023744 4591 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.026226 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:31.986934 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:32.027235 4559 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:31.928772 2597 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-78ff87b2196e4600aed5d8442f9b9bdd on tablet 78ff87b2196e4600aed5d8442f9b9bdd with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:32.027896 4591 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:32.029292 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:32.029944 2436 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:31.990154 2332 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:32.000346 3585 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:32.034587 3590 raft_consensus.cc:1226] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:31.919440 4528 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:31.957062 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.004482 2538 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.010753 2574 raft_consensus.cc:1226] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.011317 3545 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:32.036801 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.037560 3545 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.038033 2538 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.039430 2538 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.008428 4550 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.014142 4375 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:32.040071 4550 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.040282 4375 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.040367 4404 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.016598 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:32.014673 4491 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.022980 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:32.043442 4491 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:32.012917 4456 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:32.041829 3545 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.044661 4583 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.045728 3545 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.036813 4584 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:32.048624 4584 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:32.048980 4584 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:32.031378 2598 tablet_copy_service.cc:342] P fe3d17f4ad984b9088a268d58d87469c: Request end of tablet copy session cf6042e85bef47c4aac9d064d91501d1-14fa621f65da481d96d569096ded0a35 received from {username='slave'} at 127.0.0.1:56018
I20260806 03:51:32.052254 2598 tablet_copy_service.cc:433] P fe3d17f4ad984b9088a268d58d87469c: ending tablet copy session cf6042e85bef47c4aac9d064d91501d1-14fa621f65da481d96d569096ded0a35 on tablet 14fa621f65da481d96d569096ded0a35 with peer cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:32.052276 4583 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.052829 4583 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.032259 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.054219 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.032127 4475 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:51:32.054946 4477 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.055527 4475 raft_consensus.cc:3032] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:32.055919 4475 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:32.056841 4475 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:31.966740 2537 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.033334 2332 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:32.039469 4257 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.058256 4475 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.059049 2538 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.053210 4583 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.041059 4375 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.060278 3547 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.060472 4542 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.068485 4542 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:32.068663 4559 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260806 03:51:32.069905 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 103271us
I20260806 03:51:32.071185 2537 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.068907 4542 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.073148 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:32.076159 4367 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.065173 3545 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.076378 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:32.033030 4591 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:32.083717 2436 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.086814 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.066915 4582 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.088582 2538 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.089350 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.076682 4367 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.072644 4406 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.090552 2583 raft_consensus.cc:2379] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 2: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:32.091343 2538 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.069175 4559 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.093034 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.091948 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.076917 4587 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.094916 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.097328 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.097612 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.098489 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.094468 4404 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.093480 4367 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.094707 4276 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.101372 4276 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.101747 4276 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.095825 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.078945 4580 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.103397 2575 raft_consensus.cc:2379] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:32.097450 4582 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.103444 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.103506 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.104349 2436 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.082171 4456 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.108399 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.083142 4528 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:32.107775 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.121044 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.122363 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.122797 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.123525 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.110066 2572 raft_consensus.cc:2379] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 2: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:32.106776 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.110798 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.112036 2436 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.112159 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.109804 2584 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:32.126725 2583 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:32.114547 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.115193 4543 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.105463 4477 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.117549 2375 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.118031 4456 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.118680 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.084546 4591 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:32.103983 4559 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.132377 2436 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.133322 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.137589 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.136631 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.144663 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.144989 2477 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.140185 4456 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.137841 4513 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:32.149631 4513 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.150156 4513 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:32.129581 4543 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.154611 4543 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.157987 4542 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:32.142176 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.161970 2578 raft_consensus.cc:2379] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:32.144070 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.162439 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.163130 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.170213 2577 raft_consensus.cc:2379] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:32.171156 2576 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.142777 4591 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:32.113485 4580 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:32.198827 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 133732us
I20260806 03:51:32.200436 3545 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.202512 2578 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:32.203833 4582 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.204949 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:32.206079 4581 consensus_queue.cc:1046] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:32.208657 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:32.208995 4581 raft_consensus.cc:3032] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:32.209400 2480 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:32.209462 4581 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:32.204995 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:32.210898 4611 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:32.211390 4611 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:32.211750 4611 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:32.212291 4257 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.207847 2436 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.207856 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:32.170084 4412 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.214993 4412 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.216444 2436 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.217271 2332 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:32.219726 2436 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.215257 4604 raft_consensus.cc:2781] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.228899 2332 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:32.228905 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.125900 4367 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.224818 4472 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:32.215428 4412 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.225483 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.224982 4581 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:32.223188 4539 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.226218 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.234531 2332 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:32.240134 3545 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.246910 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.251842 2575 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.252223 3545 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.253137 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.256717 2332 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:32.256731 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.232457 4598 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.228487 4604 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.249766 4539 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.260895 2574 raft_consensus.cc:1226] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:32.232012 4472 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:32.220875 4513 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.221125 4491 consensus_queue.cc:1046] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260806 03:51:32.237680 4550 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.276221 2537 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.239202 2572 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:32.266520 2577 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:32.262694 4609 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.246912 4582 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:51:32.263229 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 43581.4us
I20260806 03:51:32.266459 2579 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:32.271822 3545 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.262141 4596 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:32.285944 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.263177 4604 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.259004 4598 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:32.294945 4622 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.268095 4539 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:32.306787 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 44326.7us
I20260806 03:51:32.269074 4472 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:32.272434 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.302625 2537 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.307871 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.280900 4581 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:32.272831 4584 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.266059 4618 consensus_queue.cc:1046] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260806 03:51:32.302942 4375 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.321419 2480 raft_consensus.cc:1226] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.321419 3586 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.261744 4404 consensus_queue.cc:1046] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260806 03:51:32.323520 4580 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.323801 2331 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.326063 4542 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.304507 4582 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.296116 4596 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:32.329418 4596 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:32.328998 4615 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.285301 4257 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.284946 4609 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:32.305641 4598 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.350194 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.350613 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.283077 4475 consensus_queue.cc:1046] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260806 03:51:32.358220 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.358115 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.361586 2377 raft_consensus.cc:2379] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 2: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:32.362668 2538 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.366989 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.367305 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.333022 4501 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.367018 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.309144 4622 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.372152 2476 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:32.379230 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 111503us
I20260806 03:51:32.373812 4276 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.385205 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.376652 2478 raft_consensus.cc:1226] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.376492 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:32.387259 4583 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.354554 4609 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:32.387482 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 92526.4us
I20260806 03:51:32.308861 4406 consensus_queue.cc:1046] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260806 03:51:32.402757 2436 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.406157 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:32.417471 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 93748.8us
I20260806 03:51:32.381642 4622 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.424131 4635 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.384886 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.445885 4635 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.446568 4635 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:32.455389 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 92858.9us
I20260806 03:51:32.419675 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:32.457055 2538 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.415721 4367 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:32.378122 4621 consensus_queue.cc:1046] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20260806 03:51:32.496769 2538 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:32.441587 4598 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.500216 2331 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.513397 2538 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:32.514168 4628 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260806 03:51:32.514899 2538 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.515997 2331 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:32.536741 2331 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:32.537284 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 89597.6us
I20260806 03:51:32.494382 4513 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:32.543025 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 125102us
I20260806 03:51:32.503557 4550 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.519865 4406 raft_consensus.cc:3032] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:32.462975 4456 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.552791 4650 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:32.497452 4640 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.542539 4568 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.546734 4600 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:32.538110 4615 raft_consensus.cc:2741] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260806 03:51:32.548699 4584 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.565662 2581 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.566613 2429 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.568528 4587 raft_consensus.cc:3032] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:32.573015 4474 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.573566 4491 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.553864 4406 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:32.578178 2429 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.568997 4598 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.575718 4587 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:32.586440 4276 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.587630 4638 consensus_queue.cc:1046] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260806 03:51:32.595768 4276 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.596416 4276 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.597070 4276 raft_consensus.cc:3032] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:32.597419 4276 raft_consensus.cc:729] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:32.578956 4412 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.565140 4543 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.586184 4583 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.576855 4656 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 03:51:32.630995 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 94342.7us
I20260806 03:51:32.632988 2331 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.619108 4591 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.598315 4276 consensus_queue.cc:260] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.644148 2377 raft_consensus.cc:1226] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.644629 2368 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.648720 4412 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:32.653856 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 137712us
I20260806 03:51:32.655606 2331 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.658149 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.664729 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.665252 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.665647 4656 consensus_queue.cc:1046] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:51:32.666137 2572 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:32.666771 4578 raft_consensus.cc:3032] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:32.667259 4578 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:32.667389 2583 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:32.654495 4639 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.671868 2538 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.674496 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.677317 2581 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:51:32.676463 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 132334us
I20260806 03:51:32.683614 4477 consensus_queue.cc:1046] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:32.671355 4587 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.685596 4583 consensus_queue.cc:1046] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:51:32.687673 4543 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.670410 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:32.675498 4661 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.632520 4406 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.668336 4657 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:32.678079 4559 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.518s user 0.012s sys 0.000s
I20260806 03:51:32.697158 2538 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.682201 2436 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.678779 4627 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.707504 2538 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.709040 2538 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.676903 2373 raft_consensus.cc:1226] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:32.687624 4456 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:32.720782 4412 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.742478 4412 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.743185 4412 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.722185 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:32.722892 4474 raft_consensus.cc:2726] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.675949 4578 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.751582 4578 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.724794 2538 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:32.756351 4645 consensus_queue.cc:1159] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cf6042e85bef47c4aac9d064d91501d1" responder_term: 1 status { last_received { term: 1 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 4 } }
I20260806 03:51:32.729081 4661 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:32.730226 4568 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.731681 4406 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.706334 4559 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.707058 4657 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:32.734627 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:32.736711 4600 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.734450 4662 consensus_queue.cc:1046] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:32.738088 4596 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.724747 4603 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.722362 4651 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:32.759616 2538 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:32.760839 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 78686.5us
I20260806 03:51:32.714572 4640 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:32.765226 2331 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.770777 4591 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.772285 4596 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:32.781770 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:32.782213 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 204133us
I20260806 03:51:32.783169 2436 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.773105 4539 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.779583 4600 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:32.787039 2436 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:32.779848 4640 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:32.789995 2538 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.788189 2429 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.788007 2534 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:32.782079 4603 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 03:51:32.791523 4611 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.578s user 0.009s sys 0.006s
I20260806 03:51:32.758567 4661 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.789412 4543 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:32.802498 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.793210 4640 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:32.820945 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.838128 4659 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.842006 3590 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.797055 4477 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.842458 4581 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.838737 4528 raft_consensus.cc:3032] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:32.785638 4622 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.851325 2331 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:32.855373 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 64635.7us
I20260806 03:51:32.800545 4591 consensus_queue.cc:1046] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260806 03:51:32.846122 4653 consensus_queue.cc:1046] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260806 03:51:32.799394 4596 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.052s
I20260806 03:51:32.843451 4509 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.836287 4543 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:32.850106 4528 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:32.870564 3590 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:32.871863 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.787791 4665 consensus_queue.cc:1046] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260806 03:51:32.879045 2436 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:32.876516 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.881402 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:32.890897 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 102780us
I20260806 03:51:32.857925 4671 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.896679 4664 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.854730 4660 consensus_queue.cc:1046] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260806 03:51:32.869946 4657 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.900204 4668 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:32.900859 4670 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:32.875079 4543 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.857925 4568 consensus_queue.cc:1046] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.033s
I20260806 03:51:32.876416 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:32.888065 4559 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.802670 4611 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.901379 4652 consensus_queue.cc:1046] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.111s
I20260806 03:51:32.917529 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:32.937208 4646 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.922070 4528 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:32.938936 4528 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:32.937459 4661 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.940130 4412 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.915314 4621 consensus_queue.cc:1046] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260806 03:51:32.921096 4631 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election won for term 2
W20260806 03:51:32.954663 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 75728us
I20260806 03:51:32.924507 4539 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:32.962832 4666 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260806 03:51:32.926306 4670 raft_consensus.cc:686] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:32.947046 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:32.976655 4670 consensus_queue.cc:237] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.979765 2537 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:32.963858 4604 consensus_meta.cc:287] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.656s user 0.016s sys 0.000s
W20260806 03:51:32.970237 4609 consensus_meta.cc:287] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.553s user 0.004s sys 0.010s
I20260806 03:51:32.972599 4621 raft_consensus.cc:3032] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:32.964915 4660 raft_consensus.cc:3032] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:32.947193 4622 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:32.971814 4472 consensus_meta.cc:287] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.635s user 0.015s sys 0.000s
I20260806 03:51:32.943993 4647 raft_consensus.cc:3032] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.005285 3594 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:32.998152 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:32.945354 4652 consensus_queue.cc:1046] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260806 03:51:33.001616 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:33.018467 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.018966 2583 raft_consensus.cc:1226] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:33.020328 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:33.002039 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.995234 4678 consensus_queue.cc:1046] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:33.020982 4664 raft_consensus.cc:3032] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.022217 4664 raft_consensus.cc:729] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:33.023674 4664 consensus_queue.cc:260] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.020972 4652 raft_consensus.cc:3032] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:32.988009 4631 raft_consensus.cc:686] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:33.026335 4652 raft_consensus.cc:729] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:32.988445 4635 consensus_meta.cc:287] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.541s user 0.010s sys 0.007s
I20260806 03:51:32.991135 4660 raft_consensus.cc:729] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:33.008740 4647 raft_consensus.cc:729] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.008966 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.037778 4647 consensus_queue.cc:260] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.040228 4678 raft_consensus.cc:3032] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.040799 4678 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.042725 4491 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:32.989039 4621 raft_consensus.cc:729] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.008951 4472 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.010937 3591 raft_consensus.cc:1226] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.011670 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:32.989231 4604 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:33.012574 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.013877 2572 raft_consensus.cc:1226] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:33.014478 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.015089 4580 consensus_queue.cc:1046] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:33.015942 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.005402 4659 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:33.021365 4375 raft_consensus.cc:3032] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.093111 4674 raft_consensus.cc:3032] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:33.019696 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.093803 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.032135 4611 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:33.032613 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:33.034226 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:33.034009 4635 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:32.990880 4456 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:32.922246 4676 consensus_queue.cc:1046] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.077s
I20260806 03:51:32.993157 4609 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.050773 4276 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.052980 4509 raft_consensus.cc:3032] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.057315 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.057526 4412 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.056474 4631 consensus_queue.cc:237] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.058957 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.061161 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.059602 4652 consensus_queue.cc:260] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.062770 4367 raft_consensus.cc:1050] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:51:33.063967 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.066972 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.075508 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:33.064499 4662 raft_consensus.cc:1050] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
W20260806 03:51:33.080988 4475 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.655s user 0.010s sys 0.009s
I20260806 03:51:33.088764 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.084323 4647 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.064065 4667 raft_consensus.cc:3032] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.120827 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.093696 4375 raft_consensus.cc:729] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:33.138454 2372 raft_consensus.cc:1226] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.139837 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.093168 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:33.117563 4475 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.698s user 0.012s sys 0.009s
I20260806 03:51:33.109992 4580 raft_consensus.cc:3032] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:33.147557 4609 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.117025 4509 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:33.104954 4674 raft_consensus.cc:729] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.163444 4672 raft_consensus.cc:3032] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.163915 4672 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:33.164204 4674 consensus_queue.cc:260] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.164877 4672 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.112388 4587 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.166356 4672 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.073144 4491 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.169497 4509 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.170171 4491 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:33.170948 4509 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.137261 4667 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:33.147975 4676 raft_consensus.cc:3032] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.150439 4580 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.113989 4678 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.161994 4635 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.165485 4674 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:33.157616 2484 raft_consensus.cc:1226] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.154595 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.179455 2581 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:33.191394 2436 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.150118 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.195883 2436 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.107929 4640 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.201915 2478 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.211838 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.213503 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.186074 4676 raft_consensus.cc:729] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.109051 4621 consensus_queue.cc:260] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.098953 4659 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.220777 4685 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:33.221588 4685 raft_consensus.cc:686] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.222973 4685 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.224472 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:33.142796 4660 consensus_queue.cc:260] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:33.250030 2376 raft_consensus.cc:1226] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:33.249882 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:33.235692 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.249112 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:33.185237 4474 consensus_queue.cc:1046] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260806 03:51:33.209452 4600 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:33.263069 2581 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:51:33.264246 2379 raft_consensus.cc:1226] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:33.218474 4678 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.267740 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.269620 2581 raft_consensus.cc:595] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:33.270198 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:33.272722 2578 raft_consensus.cc:2379] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:33.257486 2373 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:33.269399 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.207340 4691 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.277009 2534 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:33.277767 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.280648 4406 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:33.281201 4406 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:33.281848 4406 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.215751 4609 consensus_queue.cc:1046] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260806 03:51:33.285218 2332 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.185148 4375 consensus_queue.cc:260] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.217600 4543 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.288893 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.289386 3591 raft_consensus.cc:1226] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.296804 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.298720 2436 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:33.274480 4604 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.288143 2370 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:33.301782 2578 raft_consensus.cc:1226] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:33.302211 4640 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:33.261466 4580 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.264421 4412 consensus_queue.cc:1046] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260806 03:51:33.303889 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.304286 4622 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:33.285969 4472 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:33.281569 4584 consensus_queue.cc:1046] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:51:33.306571 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.309551 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.313334 2578 raft_consensus.cc:1226] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.335597 2478 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.324853 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.327694 2476 raft_consensus.cc:2379] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:33.320792 2332 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:33.303920 4542 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:33.333599 4690 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:33.352398 4690 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.353960 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.357813 2332 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.359192 2332 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.362272 2482 raft_consensus.cc:1226] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.304973 4676 consensus_queue.cc:260] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.364627 2482 raft_consensus.cc:1226] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.362274 4685 consensus_queue.cc:1046] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260806 03:51:33.366868 2482 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.366995 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:33.368472 2482 raft_consensus.cc:1226] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.370412 2482 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.371171 4542 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.377626 4659 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:33.330466 4539 consensus_queue.cc:1046] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260806 03:51:33.379004 2534 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:33.379961 4539 raft_consensus.cc:3032] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.380355 4622 raft_consensus.cc:686] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.329526 4581 raft_consensus.cc:3032] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.305656 4638 consensus_queue.cc:1046] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260806 03:51:33.367465 4685 raft_consensus.cc:3032] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.381125 4539 raft_consensus.cc:729] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.317430 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:33.343516 2374 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:33.354872 4472 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.400010 2332 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:33.413388 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:33.424901 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:33.396688 4581 raft_consensus.cc:729] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.409744 4685 raft_consensus.cc:729] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.394652 4528 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:33.397064 4666 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:33.413795 4622 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.458273 4690 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.460703 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.404095 4676 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.462301 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.462857 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:33.462034 4685 consensus_queue.cc:260] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.463683 2572 raft_consensus.cc:2379] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 2: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:33.465071 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.447090 4543 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.446839 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.447784 4640 consensus_queue.cc:1046] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:33.534004 4640 raft_consensus.cc:3032] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.534448 4640 raft_consensus.cc:729] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:33.449206 2429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.012s sys 0.017s
I20260806 03:51:33.446839 2583 raft_consensus.cc:1226] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:33.536995 2429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.644s user 0.012s sys 0.017s
W20260806 03:51:33.537393 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 644861us
I20260806 03:51:33.537803 4375 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.538218 2429 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.539844 4543 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:33.438226 2370 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:33.541805 2474 raft_consensus.cc:1226] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.542632 4412 raft_consensus.cc:3032] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.543111 4412 raft_consensus.cc:729] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.544771 4412 consensus_queue.cc:260] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.549098 2578 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:33.464996 2479 raft_consensus.cc:1226] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:33.519541 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 53150us
I20260806 03:51:33.432734 4513 raft_consensus.cc:3032] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.463909 2480 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.525646 3591 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.557148 2429 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:33.565322 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 165441us
I20260806 03:51:33.566251 2332 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.567700 2332 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.568753 2534 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:33.567219 2479 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.547515 4412 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.535727 4664 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.521778 4656 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:33.554461 4692 log.cc:924] Time spent T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.280s user 0.005s sys 0.000s
I20260806 03:51:33.458667 4667 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.521528 4615 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:33.558416 4513 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.530164 4406 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.606329 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:33.608685 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 162946us
I20260806 03:51:33.577029 4539 consensus_queue.cc:260] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.535573 4640 consensus_queue.cc:260] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.579118 4680 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:33.615566 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:33.589721 4611 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.579668 4685 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.645344 2583 raft_consensus.cc:2379] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:33.636077 2429 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:33.578913 2474 raft_consensus.cc:1226] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:33.633126 2484 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:33.589157 4675 raft_consensus.cc:2781] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:33.574615 4543 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.608098 2332 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.646956 2574 raft_consensus.cc:1226] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:33.559260 4583 consensus_queue.cc:1046] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
I20260806 03:51:33.564154 4475 consensus_queue.cc:1046] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260806 03:51:33.567247 4587 consensus_queue.cc:1046] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.065s
I20260806 03:51:33.641994 4640 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.649449 2478 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:33.643796 4609 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:33.643985 4639 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.673180 2482 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.650116 2429 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.595680 4604 consensus_queue.cc:1046] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260806 03:51:33.685061 4604 raft_consensus.cc:3032] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.685477 4604 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.686344 4604 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.633575 4656 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.688199 2370 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.690160 4656 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:33.690990 2584 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:33.690328 2232 catalog_manager.cc:5495] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:33.611394 4615 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.694025 4615 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.672365 4539 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.695287 4580 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:33.605214 4581 consensus_queue.cc:260] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.697139 4618 raft_consensus.cc:3032] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.697542 4618 raft_consensus.cc:729] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:33.698427 4513 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.698422 4618 consensus_queue.cc:260] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.701743 4618 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.703790 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.687901 4604 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.707008 2480 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.708503 4406 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:33.708950 4475 consensus_queue.cc:1046] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:33.651177 4653 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:33.650930 4675 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:33.710409 4675 raft_consensus.cc:3037] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.710819 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:33.711741 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 61770.2us
I20260806 03:51:33.680003 4681 consensus_queue.cc:1046] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
W20260806 03:51:33.727504 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 119495us
I20260806 03:51:33.728549 2332 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.667958 4667 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.721021 2429 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:33.676530 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 137329us
I20260806 03:51:33.720710 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:33.739737 2578 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:33.677048 4661 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.742257 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:33.743043 2373 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.714013 4513 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.719383 4653 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:51:33.752104 4694 log.cc:924] Time spent T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.176s user 0.000s sys 0.002s
I20260806 03:51:33.758597 2429 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.760722 4653 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:33.761340 4653 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:33.764230 2429 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.765717 2429 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.767907 2332 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:33.770609 2332 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.777806 4675 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.780545 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:33.782362 2332 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.785828 4631 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.746302 4600 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.790462 4475 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:33.761818 4653 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.798261 2429 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.800846 2429 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:33.782974 4611 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:51:33.803164 4509 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.632s user 0.018s sys 0.002s
I20260806 03:51:33.769390 4412 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.806046 2373 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:33.796486 4475 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.821478 4675 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:33.823359 4660 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.827914 2578 raft_consensus.cc:1226] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.831357 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:33.836350 2373 raft_consensus.cc:3037] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.788259 4622 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:33.834250 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.788056 4627 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:33.784197 4672 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:33.754992 4406 consensus_queue.cc:1046] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260806 03:51:33.831984 2534 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:33.837275 2429 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.833969 4690 consensus_queue.cc:1046] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:33.860700 4406 raft_consensus.cc:3032] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.846055 4543 consensus_queue.cc:1046] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260806 03:51:33.794700 4598 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:33.840672 2377 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:33.844561 4697 raft_consensus.cc:3032] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.826083 4691 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.845546 4584 raft_consensus.cc:3032] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:33.870189 4584 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:33.846251 2576 raft_consensus.cc:1226] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:33.847640 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 65365.9us
I20260806 03:51:33.844592 3591 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:33.842229 4676 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.852697 4688 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:33.855257 2583 raft_consensus.cc:1261] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:33.861253 4406 raft_consensus.cc:729] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:33.783277 4666 consensus_queue.cc:1046] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260806 03:51:33.874372 2332 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:33.869091 4697 raft_consensus.cc:729] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:33.893229 4672 consensus_queue.cc:1035] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:33.894165 4697 consensus_queue.cc:260] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:33.888228 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 51033.9us
I20260806 03:51:33.895843 4659 raft_consensus.cc:979] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:33.896893 4475 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:33.869182 4645 consensus_queue.cc:1046] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.028s
I20260806 03:51:33.901729 4666 raft_consensus.cc:3032] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.910889 4666 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:33.911882 4666 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.914726 4666 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.902040 4621 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.902204 4404 raft_consensus.cc:3032] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
W20260806 03:51:33.903015 4678 consensus_meta.cc:287] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.625s user 0.010s sys 0.005s
I20260806 03:51:33.878125 2373 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:33.902846 4412 consensus_queue.cc:1046] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:33.898195 4559 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:33.897995 4676 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:33.886461 4688 raft_consensus.cc:686] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:33.922169 4583 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:33.884845 4691 consensus_queue.cc:1046] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260806 03:51:33.907430 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.907492 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:33.940436 2477 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:33.904666 4672 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:33.941242 4702 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:33.910214 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:33.942252 4702 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.908427 4680 consensus_queue.cc:1046] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 03:51:33.878085 4618 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:33.916455 4404 raft_consensus.cc:729] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:33.940196 2481 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:33.937062 2534 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:33.935351 4472 consensus_queue.cc:1046] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:33.871220 4584 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.952950 4675 raft_consensus.cc:2781] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:33.953652 4675 raft_consensus.cc:686] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:33.952829 4404 consensus_queue.cc:260] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.954629 4675 consensus_queue.cc:237] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.955160 4404 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:33.957511 4688 consensus_queue.cc:237] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.949280 4583 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.959192 2374 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:33.961212 4583 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.962177 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:33.963089 2370 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
W20260806 03:51:33.962098 4703 consensus_queue.cc:1159] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "608e57d72aae496aaf8613622153aa0c" responder_term: 3 status { last_received { term: 3 index: 6 } last_committed_idx: 6 last_received_current_leader { term: 3 index: 6 } }
I20260806 03:51:33.964789 3591 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:33.965157 2572 raft_consensus.cc:1226] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:33.965587 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:33.966799 4652 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:33.968269 2233 catalog_manager.cc:5495] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 2 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:33.952286 2538 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:33.971239 2534 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:33.940166 4412 raft_consensus.cc:3032] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:33.970229 4367 consensus_queue.cc:237] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:33.974869 4412 raft_consensus.cc:729] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:33.885692 4543 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260806 03:51:33.976171 4702 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:33.946236 4697 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.980528 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:33.981361 2572 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:33.982409 4651 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:33.983048 4651 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:33.983492 3545 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:33.983711 4472 raft_consensus.cc:3032] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.980531 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:33.984820 2481 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:33.944644 4475 raft_consensus.cc:1067] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Signalling peer dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:33.979825 2480 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:33.990576 4622 raft_consensus.cc:3032] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:33.983726 4651 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.984323 4472 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:33.995033 4670 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:34.008895 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.976190 4412 consensus_queue.cc:260] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.010635 2332 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.998849 3547 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:33.962001 4584 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:34.002394 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.019345 2482 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:33.971158 4618 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.010644 4412 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:33.996897 2584 raft_consensus.cc:1226] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:34.008898 4611 consensus_queue.cc:1046] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260806 03:51:34.021456 4609 consensus_queue.cc:1046] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:34.025215 2370 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:40728
I20260806 03:51:34.025889 4587 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:34.022686 4542 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:34.010684 4627 raft_consensus.cc:3032] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:34.041209 2581 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:34.042335 4627 raft_consensus.cc:729] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.017871 4276 raft_consensus.cc:3032] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:34.004673 4702 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:34.034054 3547 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:34.006408 4583 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:33.984555 4680 consensus_queue.cc:1046] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260806 03:51:34.001987 4622 raft_consensus.cc:729] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:34.050722 4276 raft_consensus.cc:729] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:34.029460 4618 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.010716 4600 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:34.055887 4709 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:34.059206 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:34.052500 2370 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:34.043680 4627 consensus_queue.cc:260] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:33.948001 4706 consensus_queue.cc:1046] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.101s
I20260806 03:51:34.056994 4583 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.051049 4645 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.073923 2584 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:34.047088 2369 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:34.052717 4659 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.063686 4375 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:34.064287 4578 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:34.070143 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 46769.7us
I20260806 03:51:34.091141 2483 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:34.093221 3545 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.019620 4667 consensus_queue.cc:1046] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260806 03:51:34.073581 4709 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.120818 2576 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:34.037351 4639 consensus_queue.cc:1046] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260806 03:51:34.080865 2370 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:34.101410 4583 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.079857 4672 raft_consensus.cc:3032] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:34.063035 4702 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.077502 4618 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.065366 4587 raft_consensus.cc:686] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:34.090610 4528 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.120354 4661 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:34.121161 4709 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.122227 4656 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.125345 4681 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.140882 2374 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:34.125695 4456 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.113863 4645 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.118381 4578 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.130735 4672 raft_consensus.cc:729] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.131687 4627 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.075830 4472 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.143976 4611 raft_consensus.cc:3032] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:34.154040 4456 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.177769 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.178761 2373 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.180904 4542 raft_consensus.cc:3032] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:34.181864 2534 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.183014 4703 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.183296 4688 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.183712 4703 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.184336 4688 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.187083 4475 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:34.187603 4475 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.187950 4475 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.190382 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1850) took 1101 ms. Trace:
I20260806 03:51:34.190788 2483 rpcz_store.cc:276] 0806 03:51:33.088788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:33.090815 (+ 2027us) service_pool.cc:227] Handling call
0806 03:51:34.190122 (+1099307us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b54001dced4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:34.190272 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2035503872}
I20260806 03:51:34.191391 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:34.196792 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.197602 2377 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:34.199662 2534 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.200805 4491 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:34.201426 4491 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.202178 4491 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:34.151906 4656 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.205932 4667 raft_consensus.cc:3032] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:34.206521 4667 raft_consensus.cc:729] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.206039 4581 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.208062 4667 consensus_queue.cc:260] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.120613 4659 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:34.212324 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.122099 4622 consensus_queue.cc:260] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.128942 4276 consensus_queue.cc:260] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.223101 4276 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.216361 2369 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:34.235785 2534 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.134511 4618 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.181718 4542 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.136632 4528 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.217418 4688 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:34.166085 4611 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
W20260806 03:51:34.237569 4707 log.cc:924] Time spent T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.110s user 0.002s sys 0.000s
I20260806 03:51:34.208112 4375 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:34.256318 4640 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:51:34.247486 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 226944us
I20260806 03:51:34.190423 4703 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:34.262812 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 204424us
I20260806 03:51:34.249687 4656 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.153918 4681 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:34.283479 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.274088 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.233474 4404 consensus_queue.cc:1046] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260806 03:51:34.280931 4375 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.305557 2584 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:34.293131 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.308274 4670 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:34.269094 4710 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:34.269270 4587 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.297447 4578 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.266435 4667 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.324167 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.325147 2369 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:34.302664 2480 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:34.326884 2534 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.201228 4672 consensus_queue.cc:260] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.336980 4652 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:34.337590 4652 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.337632 4631 raft_consensus.cc:3032] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:34.338176 4652 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.280547 4640 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.341187 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.281016 4456 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.338311 4631 raft_consensus.cc:729] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.275516 4653 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.278084 3586 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:34.362476 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.368649 2482 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.388996 4651 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.391300 2578 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:51:34.346678 4640 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.348259 4600 raft_consensus.cc:3032] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
W20260806 03:51:34.413815 4712 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.157s user 0.002s sys 0.000s
I20260806 03:51:34.415429 4675 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.416402 4675 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.417708 4622 raft_consensus.cc:3037] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.420161 4710 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:34.420890 4710 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.415331 4456 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:34.423300 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.424268 2377 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
W20260806 03:51:34.424170 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 310658us
I20260806 03:51:34.431893 2483 raft_consensus.cc:1226] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:34.417757 4681 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.436396 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.437211 2377 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:34.439007 2480 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.447613 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:34.447834 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.403340 4688 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.449254 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:34.450054 2534 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.450600 2369 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.410617 4719 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:34.423887 4578 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.407380 4600 raft_consensus.cc:729] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.451756 2584 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:34.451762 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.456470 2583 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:34.494306 3545 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.428977 4702 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.495760 2436 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:34.498245 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 48290.8us
I20260806 03:51:34.431131 4609 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.440647 4703 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.526149 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.527112 2369 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.345911 4542 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.342123 4611 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.430954 4456 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.516523 4581 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:51:34.556730 4621 consensus_meta.cc:287] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.640s user 0.007s sys 0.012s
I20260806 03:51:34.545396 4702 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:34.517441 4675 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.517026 4659 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.579254 4406 consensus_queue.cc:260] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.579799 2370 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.509897 4578 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.585331 4656 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:34.548920 4672 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.589888 4721 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:34.590886 4404 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.551084 4528 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:34.550673 4645 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.554361 4472 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.559692 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.559410 4618 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.487180 4688 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:34.595615 4412 consensus_meta.cc:287] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.556s user 0.015s sys 0.005s
I20260806 03:51:34.505723 4719 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:34.614166 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 127271us
W20260806 03:51:34.576586 4584 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.554s user 0.016s sys 0.000s
I20260806 03:51:34.570518 4581 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.556967 4661 consensus_queue.cc:1046] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260806 03:51:34.545639 4703 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.630450 2373 raft_consensus.cc:1226] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:34.600062 4702 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:34.529373 4609 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:34.602586 4539 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.511447 4631 consensus_queue.cc:260] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.662274 2534 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.622195 4656 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:34.631718 4697 consensus_meta.cc:287] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.651s user 0.012s sys 0.007s
I20260806 03:51:34.621959 2477 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.631349 4651 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:34.680231 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48457.4us
I20260806 03:51:34.680002 4528 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.712479 4578 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:34.712683 4724 log.cc:924] Time spent T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.091s user 0.000s sys 0.001s
I20260806 03:51:34.712340 4528 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.713222 4578 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:34.713691 4578 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.714099 4578 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:34.598376 4600 consensus_queue.cc:260] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.662554 4406 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.694188 3547 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:34.695713 2374 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:34.696504 2534 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.695266 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:34.678246 4581 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:34.621351 4475 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.742762 4475 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.630976 4721 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:34.745272 4598 consensus_meta.cc:287] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.796s user 0.008s sys 0.007s
I20260806 03:51:34.746330 4721 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.634013 4404 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.697438 4539 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.715305 4412 consensus_queue.cc:1046] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
W20260806 03:51:34.730062 4491 consensus_meta.cc:287] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.526s user 0.007s sys 0.009s
I20260806 03:51:34.749778 4412 raft_consensus.cc:3032] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:34.682003 4653 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.705691 2377 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
W20260806 03:51:34.709338 4713 log.cc:924] Time spent T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
I20260806 03:51:34.751417 2370 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.714871 2483 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:34.721366 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.670584 2332 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:34.697328 4723 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:34.689275 4618 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:34.766831 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.697768 4474 raft_consensus.cc:1013] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: attempt to promote peer cf6042e85bef47c4aac9d064d91501d1: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20260806 03:51:34.683911 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 170331us
I20260806 03:51:34.697177 4656 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.691856 4702 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.734828 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:34.735138 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:34.736159 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.737859 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.790594 2482 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.738467 2583 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:34.703402 4542 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.750046 4404 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.722146 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:34.755218 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 84770.1us
W20260806 03:51:34.758611 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 62188.8us
I20260806 03:51:34.754796 2481 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:34.719816 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:34.763275 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.798135 2479 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:34.775854 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:34.730852 4651 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.778307 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:34.757933 4666 consensus_meta.cc:287] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.842s user 0.007s sys 0.011s
I20260806 03:51:34.804589 2476 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.780224 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:34.750598 4412 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:34.813323 2332 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:34.799553 2581 raft_consensus.cc:2379] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:34.768314 4697 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:34.775445 4720 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.825340 4697 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.826108 4697 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.825874 4720 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.827142 3586 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.794034 4456 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.705816 4725 consensus_queue.cc:1046] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20260806 03:51:34.805318 2576 raft_consensus.cc:3032] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:34.839144 2576 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:34.840520 4660 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.998s user 0.010s sys 0.008s
I20260806 03:51:34.840229 2576 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.841557 2576 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:34.841284 4412 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.779105 4732 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.767357 4475 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:34.857795 4475 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:34.858374 4475 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.862663 4609 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.767336 4728 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:34.816447 4528 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.817286 4688 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:34.814498 4651 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.806600 4404 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.865044 4404 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.867131 4651 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.868345 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:34.869300 2370 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.871964 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.873519 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:34.874349 2372 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.876493 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.877331 2379 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.878933 4611 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:34.776429 4618 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.882611 2476 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:34.897112 2373 raft_consensus.cc:1261] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:34.899015 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.839973 4725 raft_consensus.cc:3032] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:34.899932 2374 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:34.902827 4600 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:34.901180 4725 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:34.905148 2576 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:34.907056 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:34.907804 2476 raft_consensus.cc:2379] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:34.818970 4719 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.801565 2474 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:34.820734 4716 raft_consensus.cc:3032] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:34.819990 4615 consensus_queue.cc:1046] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:34.772588 4723 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.750519 4491 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.822746 4406 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:34.944788 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.945667 2377 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:34.846467 4664 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.795539 3542 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:34.853919 2477 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:34.856272 4732 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.818301 4645 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:34.910483 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 97242.4us
I20260806 03:51:34.955992 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:34.916203 4728 raft_consensus.cc:686] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.930640 4653 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.912477 4697 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.912807 4725 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.914529 4680 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:51:34.845299 4583 consensus_meta.cc:287] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.712s user 0.010s sys 0.008s
I20260806 03:51:34.933997 3545 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:34.981757 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 47858.6us
I20260806 03:51:34.935585 2331 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:34.918869 4375 consensus_meta.cc:287] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.614s user 0.003s sys 0.013s
I20260806 03:51:34.907575 3591 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:34.867947 4404 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.916970 4642 raft_consensus.cc:1013] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: attempt to promote peer cf6042e85bef47c4aac9d064d91501d1: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260806 03:51:34.956746 2581 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:34.956974 2332 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:34.899438 4656 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.955642 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.023277 2476 raft_consensus.cc:2379] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:51:34.932958 4710 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.511s user 0.008s sys 0.008s
I20260806 03:51:35.024377 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.024279 4710 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.026549 4710 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.027475 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.028246 3585 raft_consensus.cc:1226] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
W20260806 03:51:35.028373 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 92893.8us
I20260806 03:51:34.996326 4375 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.029707 2331 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.030663 4639 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:35.030726 4375 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.031231 4639 raft_consensus.cc:686] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.961176 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.033294 2584 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:34.963678 2578 raft_consensus.cc:2379] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:51:34.938385 4652 consensus_meta.cc:287] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.591s user 0.007s sys 0.007s
I20260806 03:51:34.962082 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.033813 3545 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:35.034464 2583 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:34.974006 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.035612 2483 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:34.938657 4659 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.931217 4716 raft_consensus.cc:729] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:34.934392 4703 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:34.935853 4412 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:34.980783 2538 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:34.948247 4528 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.911654 4640 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.993094 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:34.995685 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:34.955403 4730 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.997241 3590 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260806 03:51:34.937803 4276 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260806 03:51:34.955621 4618 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.037873 2534 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.038902 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:34.935348 4702 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.984623 4688 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:34.980464 4680 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:34.951462 4406 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.994814 4660 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:35.039775 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 82916.7us
I20260806 03:51:34.996219 4539 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.064972 2583 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.034948 4598 raft_consensus.cc:3032] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:35.046435 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.069460 2373 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:34.989547 4675 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.085927 3594 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:35.060056 2581 raft_consensus.cc:2379] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:35.038399 3545 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.060957 2368 raft_consensus.cc:1261] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:34.950321 4578 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:34.978545 4475 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.022200 4736 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:35.063215 2332 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:35.060192 4375 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:51:35.055209 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 74510.1us
I20260806 03:51:34.976157 4645 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.050547 4509 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:35.037288 4621 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.074229 4640 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.049527 4491 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:35.074815 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 165734us
I20260806 03:51:35.055195 4609 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:35.133459 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.077032 2370 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:35.077291 4656 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.078909 4406 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:34.964324 4664 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:35.087731 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 49408.9us
W20260806 03:51:35.021178 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 242190us
I20260806 03:51:35.068612 2372 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:35.092682 2583 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:35.115335 4745 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:34.985718 4583 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:34.990101 4653 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.060614 4680 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.052428 2377 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:35.058284 4528 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.122138 2538 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.060710 3593 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
W20260806 03:51:35.176442 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 54415.7us
I20260806 03:51:35.048413 4652 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.133198 4703 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260806 03:51:35.133385 4744 consensus_queue.cc:1035] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260806 03:51:35.068931 4598 raft_consensus.cc:729] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:35.054708 4659 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:35.203593 4741 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.124s user 0.001s sys 0.000s
I20260806 03:51:35.032320 4639 consensus_queue.cc:237] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.141240 3545 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:35.094615 2534 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.062785 4688 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:35.136476 4667 consensus_meta.cc:287] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.801s user 0.012s sys 0.004s
I20260806 03:51:35.120870 4375 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.142264 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:35.213554 2369 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:35.215449 3590 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.070685 4660 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.131731 4509 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.223061 4509 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.129516 4736 raft_consensus.cc:686] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:35.131400 4651 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.146947 4721 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:35.098668 4675 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.060986 4730 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:35.152591 4709 consensus_meta.cc:287] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.015s user 0.014s sys 0.003s
I20260806 03:51:35.230772 4709 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.232636 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:35.233887 3593 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:35.234855 4709 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.231909 4659 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.133968 4609 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:35.134390 4367 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:35.147256 4745 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.136961 4276 consensus_queue.cc:1046] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260806 03:51:35.260299 4745 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.134935 4621 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.262984 4621 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.151520 4604 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:35.225584 4736 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.263851 4604 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:35.264955 2482 raft_consensus.cc:1261] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:35.262984 4627 raft_consensus.cc:3032] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:35.266625 4627 raft_consensus.cc:729] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:35.184891 4697 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.185163 4723 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.185791 4732 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.186719 4720 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.188318 4734 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:35.188376 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 125240us
I20260806 03:51:35.190479 4652 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.187534 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.192561 4645 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.116292 4728 consensus_queue.cc:237] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.292284 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.187534 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.292285 2476 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.193136 4598 consensus_queue.cc:260] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.199431 4583 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.068627 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.304567 2374 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:35.209412 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.150449 4656 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
W20260806 03:51:35.211283 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 70119.6us
I20260806 03:51:35.321573 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:35.322104 4725 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:35.212131 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 117614us
I20260806 03:51:35.182672 4584 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:35.324383 4738 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:35.324612 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:35.130676 4615 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:35.184167 4748 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:35.118562 4716 consensus_queue.cc:260] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.217374 4375 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.220649 2574 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:35.221589 4578 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.222309 4653 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.241937 2436 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
W20260806 03:51:35.204308 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 52923us
I20260806 03:51:35.333798 3593 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:35.236377 4609 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.246961 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.348415 2481 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.350303 4723 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.351107 4723 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.248746 4367 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.353701 4367 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.238929 4651 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.354597 4723 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.355770 4651 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.269383 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.268296 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.356544 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.356561 2478 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:35.214875 4703 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:35.218835 4744 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:35.357999 4744 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.358392 4744 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.267110 4604 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.300280 2483 raft_consensus.cc:2379] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:51:35.301558 4581 consensus_meta.cc:287] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.549s user 0.002s sys 0.012s
W20260806 03:51:35.177920 4472 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.517s user 0.005s sys 0.009s
I20260806 03:51:35.275902 4702 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.308205 2479 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.309213 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.287856 4734 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.310961 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.386281 2372 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:35.387162 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.311937 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.312464 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.311676 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:35.317552 4662 raft_consensus.cc:1054] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Unable to promote non-voter cf6042e85bef47c4aac9d064d91501d1: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }.
Pending config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.327410 2537 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:35.391516 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 64246.1us
I20260806 03:51:35.326130 4615 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:35.330295 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:35.329656 2476 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:35.324501 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.325317 4738 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.336943 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.330017 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.337170 4748 raft_consensus.cc:686] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:35.340459 3586 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:35.353564 4697 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.356937 2583 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.255126 4730 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.277376 3590 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:35.259337 4660 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:35.360841 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 119019us
W20260806 03:51:35.369732 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 192740us
I20260806 03:51:35.440537 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.441607 2476 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:35.298431 4475 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.286083 4667 consensus_queue.cc:1046] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
I20260806 03:51:35.373833 4715 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:35.393519 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:35.457835 2579 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.401772 2537 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.403766 3547 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:35.404132 2372 raft_consensus.cc:2379] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:35.331599 4542 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:35.406865 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:35.423823 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.421032 4621 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:35.400975 4653 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.412407 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:35.466406 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.467331 2584 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.414348 2368 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:35.412436 4367 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.417398 4738 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.418128 4662 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:35.413187 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:35.307271 4722 consensus_queue.cc:1159] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cf6042e85bef47c4aac9d064d91501d1" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } }
I20260806 03:51:35.387265 4412 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:35.334527 4583 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260806 03:51:35.431494 2429 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:35.406400 4615 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.406720 4702 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:35.454566 4759 log.cc:924] Time spent T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260806 03:51:35.455127 4734 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:35.365761 4375 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:35.459726 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 56059.9us
I20260806 03:51:35.310251 2474 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:51:35.428381 4672 consensus_meta.cc:287] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.800s user 0.010s sys 0.005s
I20260806 03:51:35.412050 4651 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:35.427250 4675 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:35.408140 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 83731us
I20260806 03:51:35.468713 4542 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.471673 2332 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:35.472272 2483 raft_consensus.cc:1261] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:35.432317 4719 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:35.473248 2581 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:35.472273 2576 raft_consensus.cc:2354] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:35.474256 2484 raft_consensus.cc:1261] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:35.472808 4662 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.465474 4703 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:35.478119 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.385097 4581 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:35.481473 4528 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.515910 4581 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.511354 4667 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:35.517006 2478 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:35.517150 4667 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:35.517648 4667 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.519625 4375 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:35.489075 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 57680.5us
I20260806 03:51:35.489656 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:35.492238 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 45695.5us
I20260806 03:51:35.492659 2373 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.498883 4730 consensus_queue.cc:1035] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260806 03:51:35.494603 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.489447 2481 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
W20260806 03:51:35.498003 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 96304.4us
I20260806 03:51:35.498258 2479 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:35.499954 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:35.432255 4697 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.473352 2374 raft_consensus.cc:2362] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2. Re-sending same reply.
I20260806 03:51:35.504737 2534 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.467535 2372 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:35.495886 4604 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.502604 4406 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:35.502978 4651 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.496578 4702 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.514420 2483 raft_consensus.cc:1226] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20260806 03:51:35.515090 2480 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:35.515223 2474 raft_consensus.cc:1226] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:35.543735 3590 raft_consensus.cc:2379] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 2: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:35.545384 2537 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.520527 3586 raft_consensus.cc:1226] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:35.547389 2368 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:35.534315 4600 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.591915 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.575575 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.477698 4367 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:35.545074 2429 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.595s (SENT) [suppressed 23 similar messages]
I20260806 03:51:35.577936 2538 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.477521 4715 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.471853 4509 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:35.581395 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 84061.9us
I20260806 03:51:35.519800 4664 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.472512 4653 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.471997 4621 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.544687 2579 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
W20260806 03:51:35.582723 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 78100.5us
I20260806 03:51:35.618192 2332 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:35.527354 4716 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:35.525188 4412 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.641649 4412 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.594604 4542 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.644289 4639 consensus_queue.cc:1046] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:35.613128 4675 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.645411 4675 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.584091 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.649086 4653 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.649686 4653 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:35.650151 4653 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.456290 4748 consensus_queue.cc:237] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.588135 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.512807 4662 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:35.590699 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 45397us
I20260806 03:51:35.664191 4653 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.522320 2379 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.666754 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.669170 4412 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:35.588554 4611 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.709s user 0.008s sys 0.008s
I20260806 03:51:35.595598 4609 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.674556 4609 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.679359 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:35.597410 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.680495 2373 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:35.578768 4703 raft_consensus.cc:686] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:35.647799 4675 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.574615 4697 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.578357 2376 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.613993 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:35.584153 4456 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:35.589684 4406 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.524602 4745 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:35.623463 2584 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:35.593832 4651 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.709098 2572 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:35.711467 4702 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.599893 4600 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.600153 4375 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:35.724676 4375 raft_consensus.cc:686] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:35.724738 4627 consensus_queue.cc:260] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.726538 4600 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.632378 4766 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:35.632624 4715 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.601583 4738 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.634254 4509 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.634651 4621 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.731298 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.636904 2534 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
W20260806 03:51:35.740398 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 103581us
I20260806 03:51:35.705695 4406 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.742300 2584 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:35.742424 2534 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:35.638684 4664 consensus_queue.cc:1035] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:35.702567 4456 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.639350 4672 consensus_queue.cc:1035] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:35.639874 4472 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:35.746451 4472 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:35.747300 4472 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.626360 4706 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:35.659422 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:35.670133 4615 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.574790 4631 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:35.516582 4581 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260806 03:51:35.675463 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.675446 2368 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.677146 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:35.761834 2232 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33778 (request call id 214) took 2357 ms. Trace:
W20260806 03:51:35.762670 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 85634.1us
I20260806 03:51:35.677839 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.681970 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:35.764484 2577 raft_consensus.cc:2379] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:35.760228 4706 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.687336 2379 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.766299 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.767051 2477 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.767273 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.768532 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.768350 4706 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.695883 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.784905 2233 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33762 (request call id 223) took 1883 ms. Trace:
I20260806 03:51:35.785029 2581 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:35.696218 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:35.696623 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.697672 4412 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.697952 4688 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.691974 4716 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.706650 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.612773 4528 raft_consensus.cc:2781] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:35.786885 2574 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:35.653200 3542 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:35.788304 4528 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.788266 4716 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.789320 4528 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.791258 4716 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.740495 4734 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:35.715428 4745 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.719122 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.792228 4734 raft_consensus.cc:686] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:35.793051 2375 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 2: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:35.794269 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.726719 4375 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.796875 3545 leader_election.cc:398] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:35.797237 3545 leader_election.cc:401] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 03:51:35.727509 4766 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.730994 4738 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.636564 4367 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:35.732079 4627 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.733072 4651 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.688295 2370 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.738669 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.812793 2476 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.744441 4456 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:35.741029 4703 consensus_queue.cc:237] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:35.755119 2331 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:35.816543 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 61497.5us
I20260806 03:51:35.675120 4604 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:35.757509 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.643517 4680 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.692234 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.836472 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:35.836548 2373 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:35.837620 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.837989 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.838963 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.839288 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.839655 2574 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:35.769398 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.840454 2482 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:35.841547 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.842464 2482 raft_consensus.cc:2379] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:35.843464 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.769837 2379 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:35.845141 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.846099 2379 raft_consensus.cc:2379] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:35.770196 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.805600 4702 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:35.773025 4622 raft_consensus.cc:3032] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:35.773234 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.774763 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.776430 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.857909 2478 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.858045 2483 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.779973 4584 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:35.780732 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.761646 4675 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:35.860414 2481 raft_consensus.cc:2379] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:35.781790 2236 catalog_manager.cc:5495] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 2 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 3 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:35.784116 4615 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.799849 2375 raft_consensus.cc:2379] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:51:35.846884 4763 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.104s user 0.000s sys 0.002s
W20260806 03:51:35.801678 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 185147us
I20260806 03:51:35.802325 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.804778 3545 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.785295 2233 rpcz_store.cc:276] 0806 03:51:33.881772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:33.881967 (+ 195us) service_pool.cc:227] Handling call
0806 03:51:35.764670 (+1882703us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:34.152924 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:34.300212 (+147288us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:34.300254 (+ 42us) write_op.cc:435] Acquired schema lock
0806 03:51:34.300271 (+ 17us) tablet.cc:660] Decoding operations
0806 03:51:34.300801 (+ 530us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:34.300867 (+ 66us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:34.300894 (+ 27us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:51:34.304167 (+ 3273us) tablet.cc:699] Row locks acquired
0806 03:51:34.304186 (+ 19us) write_op.cc:260] PREPARE: finished
0806 03:51:34.304349 (+ 163us) write_op.cc:270] Start()
0806 03:51:34.304536 (+ 187us) write_op.cc:276] Timestamp: P: 1785988294304298 usec, L: 0
0806 03:51:34.304554 (+ 18us) op_driver.cc:331] REPLICATION: starting
0806 03:51:34.305379 (+ 825us) log.cc:842] Serialized 535 byte log entry
0806 03:51:35.043478 (+738099us) op_driver.cc:444] REPLICATION: finished
0806 03:51:35.328217 (+284739us) write_op.cc:301] APPLY: starting
0806 03:51:35.361981 (+ 33764us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:35.385803 (+ 23822us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:35.432716 (+ 46913us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:35.454099 (+ 21383us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:35.454285 (+ 186us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:51:35.454331 (+ 46us) write_op.cc:312] APPLY: finished
0806 03:51:35.454587 (+ 256us) log.cc:842] Serialized 23 byte log entry
0806 03:51:35.454791 (+ 204us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:35.454884 (+ 93us) write_op.cc:454] Released schema lock
0806 03:51:35.454947 (+ 63us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:35.455236 (+ 289us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:35.455271 (+ 35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":290967,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":229086,"apply.run_cpu_time_us":3259,"apply.run_wall_time_us":163932,"num_ops":1,"prepare.queue_time_us":166997,"prepare.run_cpu_time_us":4253,"prepare.run_wall_time_us":163827,"replication_time_us":729910,"thread_start_us":151097,"threads_started":1}]]}
I20260806 03:51:35.827229 2480 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.830616 4716 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:35.831436 4766 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.809514 4736 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.826963 4367 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.833488 4651 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.833904 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:35.834223 4528 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.835598 4688 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:35.864996 4675 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:35.847569 2484 raft_consensus.cc:2379] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:35.770588 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.762182 2232 rpcz_store.cc:276] 0806 03:51:33.393888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:33.394081 (+ 193us) service_pool.cc:227] Handling call
0806 03:51:35.751114 (+2357033us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:33.967762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:33.968596 (+ 834us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:33.968651 (+ 55us) write_op.cc:435] Acquired schema lock
0806 03:51:33.968669 (+ 18us) tablet.cc:660] Decoding operations
0806 03:51:33.969144 (+ 475us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:33.969260 (+ 116us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:33.969291 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:51:33.969554 (+ 263us) tablet.cc:699] Row locks acquired
0806 03:51:33.969593 (+ 39us) write_op.cc:260] PREPARE: finished
0806 03:51:33.969786 (+ 193us) write_op.cc:270] Start()
0806 03:51:33.969971 (+ 185us) write_op.cc:276] Timestamp: P: 1785988293969718 usec, L: 0
0806 03:51:33.969993 (+ 22us) op_driver.cc:331] REPLICATION: starting
0806 03:51:34.042307 (+ 72314us) log.cc:842] Serialized 535 byte log entry
0806 03:51:34.817080 (+774773us) op_driver.cc:444] REPLICATION: finished
0806 03:51:35.115075 (+297995us) write_op.cc:301] APPLY: starting
0806 03:51:35.125324 (+ 10249us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:35.134565 (+ 9241us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:35.166109 (+ 31544us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:35.205483 (+ 39374us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:35.227987 (+ 22504us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:51:35.239974 (+ 11987us) write_op.cc:312] APPLY: finished
0806 03:51:35.283647 (+ 43673us) log.cc:842] Serialized 23 byte log entry
0806 03:51:35.298681 (+ 15034us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:35.308659 (+ 9978us) write_op.cc:454] Released schema lock
0806 03:51:35.327083 (+ 18424us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:35.386062 (+ 58979us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:35.401996 (+ 15934us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":384520,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":252568,"apply.run_cpu_time_us":4479,"apply.run_wall_time_us":334909,"num_ops":1,"prepare.queue_time_us":186545,"prepare.run_cpu_time_us":42718,"prepare.run_wall_time_us":226966,"raft.queue_time_us":601584,"raft.run_cpu_time_us":25318,"raft.run_wall_time_us":374558,"replication_time_us":846948,"thread_start_us":544158,"threads_started":4,"wal-append.queue_time_us":105321}]]}
I20260806 03:51:35.847744 2476 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:35.885231 2376 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.905447 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.848229 2373 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:35.837663 4680 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:35.854724 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.903867 2479 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.908483 2379 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.855782 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.861399 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.912477 2481 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:35.913560 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.914579 2481 raft_consensus.cc:2379] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:35.915848 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.793642 4734 consensus_queue.cc:237] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:35.865213 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 320057us
I20260806 03:51:35.800252 4697 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:35.765434 4744 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.930738 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.931524 2484 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:35.883751 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:35.891186 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 57362.9us
I20260806 03:51:35.933020 2368 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:35.933321 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:35.885283 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 80634.1us
I20260806 03:51:35.857363 4702 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:35.933645 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:35.860818 4584 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:35.910887 4675 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:35.915477 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:35.887044 4716 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.923785 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.926465 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.889880 4618 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:35.899849 4706 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:35.902827 4714 raft_consensus.cc:3032] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:35.930183 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:35.931548 2429 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:35.933585 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:35.934948 2331 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.918500 4728 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:35.979251 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:35.931146 4697 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:35.984483 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:36.000733 3594 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:35.841651 4738 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:35.939488 2577 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.959403 2482 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:35.960103 2476 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:35.944494 2481 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.956429 4766 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.958700 4716 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:35.913136 4367 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.925203 2474 raft_consensus.cc:2379] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:35.954555 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:36.025377 2373 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:36.025636 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:35.972273 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.973246 4706 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:35.978608 4600 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.975092 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:35.976409 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:36.034556 2368 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.026371 3591 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:35.990037 4662 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.903452 4688 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.936583 3545 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:35.938664 2477 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:35.944738 4702 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:35.930796 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:35.983651 2480 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:35.912254 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:35.943599 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:35.945734 4680 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:35.981076 4721 consensus_meta.cc:287] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.736s user 0.011s sys 0.005s
I20260806 03:51:35.966359 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.069204 2581 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:36.006594 2331 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:36.027958 4766 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:35.968003 4722 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:36.027678 2376 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:35.993768 4728 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.029345 2478 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:36.029887 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:36.084535 2484 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:36.030684 4276 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.972179 4714 raft_consensus.cc:729] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:35.971838 4651 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:35.989564 2574 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:36.036616 2436 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
W20260806 03:51:36.095003 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 58492.2us
I20260806 03:51:35.982167 4609 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:36.059011 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:36.039935 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 103464us
W20260806 03:51:36.046217 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 114780us
I20260806 03:51:35.997627 4672 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:35.999190 4697 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:35.998035 4744 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.046871 2379 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:35.957604 4667 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.054561 2373 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:36.053751 2481 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:36.063812 2576 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:36.070305 2368 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:36.115453 2436 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:36.005138 2479 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:36.118659 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.038398 4706 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:36.080996 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 74502.6us
I20260806 03:51:36.082374 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.050637 4581 consensus_queue.cc:1046] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.042s
I20260806 03:51:36.052161 4583 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:36.090080 2577 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:36.063203 4615 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:36.037364 2579 raft_consensus.cc:2354] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
W20260806 03:51:36.038396 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 417187us
I20260806 03:51:36.202687 2436 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:36.204797 2331 leader_election.cc:398] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.205323 2331 leader_election.cc:401] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 03:51:36.206117 2538 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:36.120242 2534 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:36.208453 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 88331.7us
I20260806 03:51:36.121071 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.123677 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:36.211511 2479 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:35.861040 4622 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:36.137382 4771 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:36.212687 4664 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:36.212775 4771 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:36.138765 4744 consensus_queue.cc:1035] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:36.213446 4771 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.137820 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.139930 4367 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.142630 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:36.143389 4680 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:36.148389 4528 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.081461 3585 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:36.232417 2332 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:36.150194 4662 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.169514 4688 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.117921 4609 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:36.215392 4631 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:36.233968 4609 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:36.234704 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:36.172338 2374 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:36.237701 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.173693 4456 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.176188 4598 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:36.181777 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.188566 4539 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:36.130539 4651 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.132273 4621 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:36.211621 2373 raft_consensus.cc:2379] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:36.212965 4730 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:36.214725 2538 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:36.144170 4639 consensus_queue.cc:1046] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:36.076228 4721 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:36.214915 2534 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
W20260806 03:51:36.273810 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 59044.2us
W20260806 03:51:36.276980 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 62364.7us
I20260806 03:51:36.278409 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:36.245914 4666 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:36.219035 2579 raft_consensus.cc:1261] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:51:36.228773 2429 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:36.213407 4664 raft_consensus.cc:686] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:36.230755 4680 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.233273 4509 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:36.246568 2577 raft_consensus.cc:1261] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:36.248835 2479 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:36.264112 4627 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:36.238205 4583 raft_consensus.cc:686] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:36.083119 4715 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:36.259925 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:36.261350 2578 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.018s sys 0.111s
W20260806 03:51:36.303900 2578 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s user 0.018s sys 0.111s
W20260806 03:51:36.128680 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 232161us
I20260806 03:51:36.213771 4615 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.271185 2583 raft_consensus.cc:2354] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:36.047750 4600 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.284974 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:36.285490 4622 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.266902 4539 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:36.269271 2375 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:36.286401 2538 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:36.276160 4730 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:36.315603 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:36.316154 2538 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:36.287304 4743 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:36.287885 4688 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:36.284842 4744 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
W20260806 03:51:36.291322 4406 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.550s user 0.011s sys 0.001s
I20260806 03:51:36.291900 4772 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:36.294232 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:36.336680 2534 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:36.299041 2483 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:36.262053 4412 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:36.083279 4719 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:36.318044 4730 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:36.295264 4621 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.222477 4472 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.317694 2537 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:36.214619 4722 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:36.317706 2538 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:36.322235 4748 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.022s sys 0.072s
I20260806 03:51:36.305764 4715 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:36.325115 4375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.030s sys 0.027s
I20260806 03:51:36.317677 4539 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:36.384033 4748 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s user 0.022s sys 0.073s
W20260806 03:51:36.389027 4375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.586s user 0.031s sys 0.027s
I20260806 03:51:36.320791 4528 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:36.360488 4584 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.321458 4662 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:36.394901 4584 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:36.362054 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 44460.1us
I20260806 03:51:36.396212 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:36.396929 2375 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.397177 4600 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:36.397321 2537 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:36.329119 2377 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:36.398056 4583 consensus_queue.cc:237] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.399222 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 890) took 1090 ms. Trace:
I20260806 03:51:36.214983 4672 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.310753 4627 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.401325 4615 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:36.401443 4672 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.303148 4666 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.389130 4715 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:36.389292 4738 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:36.404930 4667 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.405161 4651 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.384268 4367 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:36.405593 4666 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.402145 2370 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:36.406569 4651 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:36.390051 4719 raft_consensus.cc:686] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:36.407959 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.408475 4472 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.408736 2370 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:36.408898 4719 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.409198 4664 consensus_queue.cc:237] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.409787 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:36.410566 2370 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.401341 4722 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.355535 4745 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:36.399569 2377 rpcz_store.cc:276] 0806 03:51:35.308597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:35.308773 (+ 176us) service_pool.cc:227] Handling call
0806 03:51:36.399041 (+1090268us) spinlock_profiling.cc:243] Waited 838 ms on lock 0x7b54001eaad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:36.399157 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1926491392}
I20260806 03:51:36.412823 4745 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:36.413280 4745 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.413537 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:36.377802 4767 log.cc:924] Time spent T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.262s user 0.001s sys 0.000s
I20260806 03:51:36.361670 4622 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.382579 4631 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.428133 2578 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260806 03:51:36.330291 4581 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:36.416525 2374 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:36.418963 2534 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:36.420351 3594 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.351539 4744 raft_consensus.cc:686] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:36.401770 4627 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.414165 2377 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 4
I20260806 03:51:36.391992 4748 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:36.380236 4766 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.385784 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:36.392131 4276 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:36.379163 4412 raft_consensus.cc:686] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:36.403692 4738 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.349227 4406 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.446014 2534 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:36.447614 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.447835 3591 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:36.451661 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:36.423328 4584 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:36.432452 4618 raft_consensus.cc:686] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:36.352461 4772 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.430540 4728 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.413300 3542 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c
W20260806 03:51:36.433352 4703 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.014s sys 0.019s
I20260806 03:51:36.443408 4748 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:36.444113 4375 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:36.454099 4631 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:36.483513 4766 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:36.510557 2583 raft_consensus.cc:2354] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:36.432232 4675 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:36.523882 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 77995.5us
I20260806 03:51:36.441509 4581 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.569610 2373 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.423355 4714 consensus_queue.cc:260] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.532531 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.585994 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:36.535588 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:36.524976 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 322376us
I20260806 03:51:36.537477 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:36.588135 3593 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:36.499125 4456 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.542218 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.544421 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.510471 4680 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:36.567708 4748 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.536139 4528 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:51:36.568176 2482 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.002s sys 0.034s
I20260806 03:51:36.568259 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:36.568584 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:36.601388 2482 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s user 0.003s sys 0.034s
I20260806 03:51:36.590601 2436 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:36.586755 2578 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:36.539481 4651 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:36.600514 2236 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.699s user 0.017s sys 0.023s
W20260806 03:51:36.560710 4703 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s user 0.014s sys 0.019s
W20260806 03:51:36.603962 2236 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.702s user 0.017s sys 0.023s
I20260806 03:51:36.601926 2331 leader_election.cc:398] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:36.606307 2331 leader_election.cc:401] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 03:51:36.607205 2574 raft_consensus.cc:2354] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.611586 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:36.612309 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:36.612308 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.612537 2483 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:36.544605 4734 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260806 03:51:36.568981 4375 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:36.536011 4584 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.620585 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.618562 3545 leader_election.cc:398] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:36.621526 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:36.625914 3545 leader_election.cc:401] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 03:51:36.613287 2481 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:36.622462 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:36.627396 2576 raft_consensus.cc:2354] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:36.628768 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:36.630664 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.631454 2480 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:36.635070 2481 raft_consensus.cc:2354] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.568365 4766 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:36.598603 4728 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:36.640194 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.606406 4528 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:36.635864 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.641090 2476 raft_consensus.cc:2379] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:36.652976 2480 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:36.642007 2332 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:36.636276 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:36.654717 2481 raft_consensus.cc:2354] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.645078 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:36.655691 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 427901us
I20260806 03:51:36.647658 2484 raft_consensus.cc:1261] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:51:36.656448 2332 leader_election.cc:398] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.656504 2429 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:36.647672 2483 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:51:36.617035 4276 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:36.658733 2331 leader_election.cc:398] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.659173 3592 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:36.624660 4714 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.667829 2332 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:36.665730 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.665459 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.623766 4375 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:36.666283 2331 leader_election.cc:401] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 03:51:36.669749 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.545921 4744 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.673943 2332 leader_election.cc:398] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.678057 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.677084 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.680506 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.641160 4627 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:36.694717 2331 leader_election.cc:398] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:36.699093 2331 leader_election.cc:401] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 03:51:36.695878 2480 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:36.608654 4722 consensus_queue.cc:1035] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.052s
I20260806 03:51:36.613534 4680 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.058s
I20260806 03:51:36.696637 2429 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:36.699575 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.662420 4528 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:36.665827 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.697715 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:36.658411 4766 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:36.696238 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:36.702337 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.661729 4782 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:36.573374 4631 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:36.612887 4651 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.705065 2476 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:36.696395 2481 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:36.669313 4666 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:36.824714 2373 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:36.693929 4672 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:51:36.713202 4716 consensus_meta.cc:287] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.680s user 0.010s sys 0.009s
I20260806 03:51:36.846367 4672 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:36.846812 4598 consensus_meta.cc:287] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.584s user 0.015s sys 0.006s
I20260806 03:51:36.846928 4672 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:36.847640 4604 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:36.847895 2372 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:36.848290 4604 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.851060 4604 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:36.859196 4776 log.cc:924] Time spent T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20260806 03:51:36.861775 4584 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.701996 4615 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:36.665992 4276 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:51:36.747143 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 68832.3us
I20260806 03:51:36.868276 4276 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.869398 2332 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:36.871328 4472 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:36.871719 4472 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:36.872072 4472 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:36.805956 4771 consensus_meta.cc:287] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.592s user 0.023s sys 0.000s
W20260806 03:51:36.790524 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 75441.2us
I20260806 03:51:36.877467 2331 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:36.609020 4412 consensus_queue.cc:237] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.797523 2377 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:36.696182 4621 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:36.636116 4703 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
I20260806 03:51:36.665442 4667 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:36.798843 3594 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:36.662606 4728 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.699609 4790 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:36.808616 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1610) took 1004 ms. Trace:
I20260806 03:51:36.897140 2482 rpcz_store.cc:276] 0806 03:51:35.804310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.003292 (+198982us) service_pool.cc:227] Handling call
0806 03:51:36.017077 (+ 13785us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:36.017593 (+ 516us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:36.017631 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:36.019347 (+ 1716us) log.cc:842] Serialized 548 byte log entry
0806 03:51:36.602402 (+583055us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:36.602447 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:36.602522 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:36.808335 (+205813us) raft_consensus.cc:1671] finished
0806 03:51:36.808358 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:36.808545 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":582491,"threads_started":1,"wal-append.queue_time_us":650846}
W20260806 03:51:36.789757 4702 consensus_meta.cc:287] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.721s user 0.009s sys 0.006s
W20260806 03:51:36.879371 4774 log.cc:924] Time spent T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.375s user 0.004s sys 0.004s
I20260806 03:51:36.811110 2480 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:36.816817 2479 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:36.797701 4791 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:36.665751 4618 consensus_queue.cc:237] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.834748 2476 raft_consensus.cc:1261] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:36.834848 2375 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:36.825225 4666 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:36.837911 4706 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.647s user 0.014s sys 0.000s
I20260806 03:51:36.845592 4675 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:36.846561 4716 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:36.744618 4406 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:36.660619 4456 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:36.874363 4581 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.874963 4772 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.875420 4792 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:36.798146 4600 consensus_queue.cc:1035] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260806 03:51:36.815485 2478 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:36.950999 2537 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:36.885447 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1593) took 1082 ms. Trace:
W20260806 03:51:36.812145 2474 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.015s sys 0.024s
W20260806 03:51:36.886070 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 189536us
I20260806 03:51:36.952148 2377 rpcz_store.cc:276] 0806 03:51:35.803110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.413256 (+610146us) service_pool.cc:227] Handling call
0806 03:51:36.885366 (+472110us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:36.952831 2429 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:36.886590 2332 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:36.887418 2331 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:36.889055 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.889719 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.953660 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:36.891242 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.025s sys 0.078s
I20260806 03:51:36.954733 2377 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
W20260806 03:51:36.955076 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.025s sys 0.078s
W20260806 03:51:36.955909 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 638268us
I20260806 03:51:36.957470 2538 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:36.893253 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.958652 2572 raft_consensus.cc:2354] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:36.895478 2574 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:36.811108 3593 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:36.816120 4782 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:36.819978 4738 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.959812 4782 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:36.960126 2474 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s user 0.016s sys 0.024s
I20260806 03:51:36.961573 4456 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.918129 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:51:36.965557 4788 log.cc:924] Time spent T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20260806 03:51:36.965808 2534 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:36.969179 4456 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:36.969048 4782 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.971362 4782 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:36.973481 4688 consensus_meta.cc:287] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.648s user 0.012s sys 0.000s
I20260806 03:51:36.973933 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1612) took 1069 ms. Trace:
I20260806 03:51:36.973940 4688 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:36.976913 4716 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.974574 2474 rpcz_store.cc:276] 0806 03:51:35.904321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.031383 (+127062us) service_pool.cc:227] Handling call
0806 03:51:36.178091 (+146708us) spinlock_profiling.cc:243] Waited 5.18 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:36.178509 (+ 418us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:36.179113 (+ 604us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:36.179159 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:36.179986 (+ 827us) log.cc:842] Serialized 26 byte log entry
0806 03:51:36.960668 (+780682us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:36.960720 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:36.960857 (+ 137us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:36.973442 (+ 12585us) raft_consensus.cc:1671] finished
0806 03:51:36.973487 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:36.973847 (+ 360us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":22500,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":11939,"spinlock_wait_cycles":11904256,"thread_start_us":676863,"threads_started":1,"wal-append.queue_time_us":698733}
I20260806 03:51:36.977716 2368 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:36.979848 2436 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:36.918457 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.918941 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:36.980638 2429 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:36.927240 2575 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:36.891122 4680 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:36.811650 4715 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.894797 4721 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
W20260806 03:51:36.891230 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 478002us
I20260806 03:51:36.938259 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1581) took 1152 ms. Trace:
I20260806 03:51:37.019904 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:37.020385 2373 rpcz_store.cc:276] 0806 03:51:35.785458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.293895 (+508437us) service_pool.cc:227] Handling call
0806 03:51:36.937982 (+644087us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b54001c0a54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:36.938170 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":550297856}
I20260806 03:51:36.890779 4771 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.984153 2538 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:36.984686 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 97383.6us
I20260806 03:51:36.984769 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:36.915555 4702 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:36.985925 4792 raft_consensus.cc:686] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:36.929039 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:36.986534 4581 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:36.988205 2429 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:37.039620 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 51513.8us
I20260806 03:51:37.040699 2429 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.042297 2429 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:36.988545 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 102037us
I20260806 03:51:36.930258 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 123) took 1580 ms. Trace:
I20260806 03:51:37.044396 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.044642 2375 rpcz_store.cc:276] 0806 03:51:35.349530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.360665 (+1011135us) service_pool.cc:227] Handling call
0806 03:51:36.926776 (+566111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.045647 3590 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.993386 4716 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:37.053135 4716 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:37.053643 4716 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.054473 4702 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.055222 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:37.046775 4672 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.058683 4672 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.060473 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.061303 2572 raft_consensus.cc:2354] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:37.061873 4472 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.063791 4728 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:36.998360 4675 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:37.006628 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40712 (request call id 489) took 1698 ms. Trace:
W20260806 03:51:37.009366 4539 consensus_meta.cc:287] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.618s user 0.019s sys 0.011s
I20260806 03:51:37.093587 4539 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.096261 3542 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.010265 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1586) took 1209 ms. Trace:
I20260806 03:51:37.014539 4615 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:36.988348 4667 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.097478 4667 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:37.016734 4662 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.623s user 0.012s sys 0.000s
I20260806 03:51:37.017725 4808 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.017395 4528 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.018342 4703 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:51:37.018635 3586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36484 (request call id 253) took 1046 ms. Trace:
I20260806 03:51:37.020896 3592 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:36.926841 4666 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:36.917821 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:37.023339 2534 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:37.129849 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 106652us
W20260806 03:51:37.030381 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 46335.8us
I20260806 03:51:37.132618 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:37.134866 2538 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:37.033288 2377 raft_consensus.cc:2379] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:37.034334 2331 leader_election.cc:398] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:37.036577 2379 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:37.136562 2331 leader_election.cc:401] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:51:37.035720 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.137226 2373 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.137323 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.138559 2377 raft_consensus.cc:2379] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:37.139537 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.044464 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.048640 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.144265 2481 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:37.144677 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 100313us
I20260806 03:51:36.988788 4772 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:37.049809 2429 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:37.150871 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 101176us
W20260806 03:51:37.004101 2372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 122) took 1654 ms (client timeout 1712 ms). Trace:
I20260806 03:51:36.933121 4706 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.001215 4807 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:37.001428 4790 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:37.081653 2477 raft_consensus.cc:2354] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:37.083704 4631 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.038252 4688 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:37.083952 4791 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:37.068656 4604 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:37.066404 4621 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:37.085021 4806 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:37.064924 4792 consensus_queue.cc:237] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.168190 4600 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:37.091029 4651 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:37.092208 3545 leader_election.cc:398] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:37.218456 3545 leader_election.cc:401] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 03:51:37.075673 2369 rpcz_store.cc:276] 0806 03:51:35.308203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:35.308915 (+ 712us) service_pool.cc:227] Handling call
0806 03:51:37.005898 (+1696983us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54000127d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4aaf0 00007f0d7cacdf2a 00007f0d81a0ac6e 00007f0d7b101807 00007f0d7b101792 00007f0d7b1016c1 00007f0d7b10163c 00007f0d7b1008f2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.006520 (+ 622us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3412515584}
W20260806 03:51:37.051185 4736 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.162s user 0.018s sys 0.001s
I20260806 03:51:36.963004 4276 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.142478 2436 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:36.995786 4664 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:37.146355 2377 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:51:37.152220 2372 rpcz_store.cc:269] 0806 03:51:35.349212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:35.587667 (+238455us) service_pool.cc:227] Handling call
0806 03:51:37.003871 (+1416204us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54000127d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.004011 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2746889984}
I20260806 03:51:37.097139 2368 rpcz_store.cc:276] 0806 03:51:35.800449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.137443 (+336994us) service_pool.cc:227] Handling call
0806 03:51:37.009977 (+872534us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54003e99d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.010178 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1754246912}
W20260806 03:51:37.156440 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 122235us
I20260806 03:51:37.157042 2332 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:37.118398 4609 consensus_meta.cc:287] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.884s user 0.007s sys 0.009s
I20260806 03:51:37.163156 2373 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:37.164688 2579 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:37.113606 4662 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:51:37.067911 4697 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.843s user 0.010s sys 0.008s
I20260806 03:51:37.116628 4675 raft_consensus.cc:686] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:37.103698 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 53834us
W20260806 03:51:37.163808 2370 consensus_meta.cc:287] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.753s user 0.015s sys 0.001s
I20260806 03:51:37.251513 2370 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
W20260806 03:51:37.252331 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 100442us
I20260806 03:51:37.252677 4697 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.253221 2429 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.254441 2429 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.255384 4697 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.255539 2429 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.256963 2483 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:37.257814 2429 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.252417 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1601) took 1295 ms. Trace:
I20260806 03:51:37.259946 2370 rpcz_store.cc:276] 0806 03:51:35.957381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.409493 (+452112us) service_pool.cc:227] Handling call
0806 03:51:37.252334 (+842841us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.224996 4621 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:37.122400 4472 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.198764 2374 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:37.194624 2481 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:51:37.149888 4772 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:37.156253 4615 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:37.272583 4772 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:37.272737 4615 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:37.273260 4615 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:37.164330 4672 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.280853 3585 raft_consensus.cc:2379] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:37.280426 4672 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.249182 4664 raft_consensus.cc:686] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:37.282743 4664 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.157693 4456 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:37.285844 4276 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:37.286736 4276 raft_consensus.cc:2741] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 03:51:37.156349 4581 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:37.158666 4734 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:37.089596 4375 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.165230 4702 raft_consensus.cc:2741] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:51:37.166548 4807 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:37.200328 4790 raft_consensus.cc:686] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:51:37.215631 2379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 126) took 1616 ms (client timeout 2093 ms). Trace:
W20260806 03:51:37.303776 2379 rpcz_store.cc:269] 0806 03:51:35.599480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.118282 (+518802us) service_pool.cc:227] Handling call
0806 03:51:37.215391 (+1097109us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b5400438fd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.215571 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2108292864}
I20260806 03:51:37.287581 4276 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:37.168874 4367 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:51:37.220821 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 128809us
I20260806 03:51:37.120323 4808 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.290971 4621 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:37.225139 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1597) took 1344 ms. Trace:
W20260806 03:51:37.226647 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 69743.9us
I20260806 03:51:37.228088 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:37.229072 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.314518 2377 rpcz_store.cc:276] 0806 03:51:35.881228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.139230 (+1258002us) service_pool.cc:227] Handling call
0806 03:51:37.225080 (+ 85850us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.315323 2584 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.229797 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:37.230365 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:37.229851 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.205577 4539 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.231994 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1577) took 1449 ms. Trace:
I20260806 03:51:37.322549 2373 rpcz_store.cc:276] 0806 03:51:35.782431 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.035304 (+1252873us) service_pool.cc:227] Handling call
0806 03:51:37.231919 (+196615us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.323720 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.236579 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:37.231994 4600 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:37.237424 4771 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:37.238580 4800 log.cc:924] Time spent T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.217s user 0.001s sys 0.000s
I20260806 03:51:37.244177 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.249241 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.197193 3590 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:37.262306 4778 raft_consensus.cc:3032] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:37.226117 4736 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:37.221381 4744 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:37.221728 4667 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.270352 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.122545 3586 rpcz_store.cc:276] 0806 03:51:35.972527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:35.978299 (+ 5772us) service_pool.cc:227] Handling call
0806 03:51:35.989354 (+ 11055us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:36.370716 (+381362us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:36.370771 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:36.371548 (+ 777us) log.cc:842] Serialized 26 byte log entry
0806 03:51:36.744172 (+372624us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:36.744248 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:36.744345 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:37.018353 (+274008us) raft_consensus.cc:1671] finished
0806 03:51:37.018376 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:37.018564 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":41314,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":91,"thread_start_us":695694,"threads_started":2,"wal-append.queue_time_us":523311}
I20260806 03:51:37.264256 4716 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.383446 4615 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.386418 4716 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:37.278692 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:37.387435 3584 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:37.165894 4406 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:37.359877 4672 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.301255 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1596) took 1440 ms. Trace:
I20260806 03:51:37.305133 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:37.196875 4688 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:37.288833 4715 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:37.264442 4604 raft_consensus.cc:686] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:51:37.223454 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 81137.2us
I20260806 03:51:37.367892 4706 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:37.294771 4743 consensus_meta.cc:287] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.963s user 0.014s sys 0.009s
I20260806 03:51:37.285897 4456 raft_consensus.cc:686] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:37.252488 3542 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:37.326296 2369 raft_consensus.cc:2379] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:37.410159 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1606) took 1199 ms. Trace:
I20260806 03:51:37.410552 2369 rpcz_store.cc:276] 0806 03:51:36.210921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.219847 (+1008926us) service_pool.cc:227] Handling call
0806 03:51:37.410087 (+190240us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.326929 2534 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:37.411767 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 84944us
I20260806 03:51:37.328107 2332 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:51:37.322062 4627 consensus_meta.cc:287] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.620s user 0.010s sys 0.005s
W20260806 03:51:37.330153 4813 log.cc:924] Time spent T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260806 03:51:37.322954 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:37.413182 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 85217.8us
I20260806 03:51:37.415946 2377 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:37.399415 4688 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:37.333349 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.418401 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:37.418740 2574 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.335927 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.419209 2369 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:37.304771 4807 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.421290 3590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36484 (request call id 252) took 1488 ms. Trace:
I20260806 03:51:37.362597 2584 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:37.363682 2578 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:37.411281 2370 raft_consensus.cc:2354] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:37.368499 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:51:37.359695 4745 consensus_meta.cc:287] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.946s user 0.019s sys 0.003s
I20260806 03:51:37.402842 4406 raft_consensus.cc:686] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:37.370039 2373 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:37.368346 4584 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
W20260806 03:51:37.374132 4820 log.cc:924] Time spent T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.108s user 0.001s sys 0.000s
I20260806 03:51:37.365768 4675 consensus_queue.cc:237] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:37.369494 4748 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.756s user 0.013s sys 0.000s
I20260806 03:51:37.348454 4528 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:37.387914 4806 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:37.358856 4621 raft_consensus.cc:686] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:37.263280 4412 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:37.392216 2368 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:51:37.393214 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 258434us
I20260806 03:51:37.472630 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1602) took 1510 ms. Trace:
I20260806 03:51:37.473016 2368 rpcz_store.cc:276] 0806 03:51:35.962577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.225774 (+1263197us) service_pool.cc:227] Handling call
0806 03:51:37.472555 (+246781us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.473449 2538 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:37.474154 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.387513 4615 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.395026 2379 raft_consensus.cc:2354] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3.
I20260806 03:51:37.394639 2374 rpcz_store.cc:276] 0806 03:51:35.872875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.447307 (+574432us) service_pool.cc:227] Handling call
0806 03:51:37.301101 (+853794us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b54000c0654. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.301201 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1586113792}
I20260806 03:51:37.496729 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.497433 2374 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.496995 4456 consensus_queue.cc:237] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.248704 4609 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.500892 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:37.501550 2576 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:37.501699 4609 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.367995 4778 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:37.502815 4778 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.503479 4406 consensus_queue.cc:237] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.504596 4778 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.408633 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:37.416437 2332 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.506418 3593 raft_consensus.cc:2354] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:37.416585 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1579) took 1632 ms. Trace:
W20260806 03:51:37.506916 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 90567.7us
I20260806 03:51:37.507292 2377 rpcz_store.cc:276] 0806 03:51:35.783907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.322517 (+1538610us) service_pool.cc:227] Handling call
0806 03:51:37.416533 (+ 94016us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.332120 2575 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:37.508548 2436 leader_election.cc:398] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:37.508433 4375 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:37.509018 2436 leader_election.cc:401] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 03:51:37.508896 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.420449 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.415438 4797 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:37.509815 4472 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:37.510288 4797 raft_consensus.cc:686] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:37.510520 4528 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.510604 4412 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:37.511261 4528 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.510910 4615 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260806 03:51:37.511873 4797 consensus_queue.cc:237] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.514158 4615 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:37.321830 4730 consensus_meta.cc:287] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.933s user 0.016s sys 0.007s
I20260806 03:51:37.511874 2332 leader_election.cc:398] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
W20260806 03:51:37.515112 4822 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20260806 03:51:37.515417 4730 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.516235 2332 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:37.422703 2331 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:37.423421 2534 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:37.431207 3592 raft_consensus.cc:2354] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:37.421749 3590 rpcz_store.cc:276] 0806 03:51:35.933140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:35.933302 (+ 162us) service_pool.cc:227] Handling call
0806 03:51:37.421063 (+1487761us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400541354. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.421183 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2466068864}
W20260806 03:51:37.517477 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 94861.5us
I20260806 03:51:37.431964 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:37.517853 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 94496.2us
I20260806 03:51:37.518287 2584 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:37.432492 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1584) took 1638 ms. Trace:
I20260806 03:51:37.519034 2373 rpcz_store.cc:276] 0806 03:51:35.794394 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.323430 (+1529036us) service_pool.cc:227] Handling call
0806 03:51:37.432432 (+109002us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.410204 4743 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:37.434684 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:37.523927 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.523993 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.524358 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.524627 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.524976 2481 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:51:37.525309 4662 raft_consensus.cc:2781] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:37.525564 2476 raft_consensus.cc:2396] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 2 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260806 03:51:37.525810 4662 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:37.526448 4662 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:37.439776 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 124) took 2090 ms. Trace:
I20260806 03:51:37.527616 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:37.527894 4715 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:37.524718 2478 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.528045 2370 rpcz_store.cc:276] 0806 03:51:35.349820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.263213 (+1913393us) service_pool.cc:227] Handling call
0806 03:51:37.434274 (+171061us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.528800 4715 raft_consensus.cc:686] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:37.529609 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.530516 2377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:51:37.530247 4715 consensus_queue.cc:237] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.532502 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.434628 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.442667 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.415429 4627 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
W20260806 03:51:37.452720 4823 log.cc:924] Time spent T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260806 03:51:37.552662 2572 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:37.416873 4790 consensus_queue.cc:237] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.424396 4721 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:37.412725 4830 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:37.538394 2478 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:37.438499 4618 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260806 03:51:37.455854 4604 consensus_queue.cc:237] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:37.421798 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 169424us
I20260806 03:51:37.430999 3585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 289) took 1158 ms. Trace:
I20260806 03:51:37.394876 4744 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.418965 4706 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:51:37.482434 4766 consensus_meta.cc:287] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.656s user 0.013s sys 0.000s
I20260806 03:51:37.580065 4766 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.482985 4808 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:37.485983 4714 consensus_meta.cc:287] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.810s user 0.007s sys 0.006s
I20260806 03:51:37.433984 4745 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.585748 4730 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:37.586983 4745 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:37.488302 4539 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.491438 4621 consensus_queue.cc:237] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.465266 4748 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:37.597440 4539 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.431557 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:37.493988 4831 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:37.495021 2538 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:37.616616 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 121713us
I20260806 03:51:37.519534 2534 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:37.519752 3545 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:37.619108 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 99663us
W20260806 03:51:37.619284 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 99611.3us
I20260806 03:51:37.520401 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.520807 2576 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260806 03:51:37.522765 4728 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:37.564148 4792 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:37.441382 4367 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:37.444283 4814 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.126s user 0.001s sys 0.000s
I20260806 03:51:37.450739 4666 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.536312 2373 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:37.536558 2369 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 4
I20260806 03:51:37.542580 2331 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:37.542923 2374 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:37.524988 2379 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:37.548452 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.548717 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.592054 4706 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:37.609611 4806 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.600765 4714 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.514894 4528 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:37.615561 4672 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.564098 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.567164 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.568629 2368 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:37.687757 2483 raft_consensus.cc:2396] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:37.688086 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1605) took 1477 ms. Trace:
I20260806 03:51:37.628288 4730 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:37.688441 2368 rpcz_store.cc:276] 0806 03:51:36.210595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.473854 (+1263259us) service_pool.cc:227] Handling call
0806 03:51:37.687989 (+214135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.633205 4728 raft_consensus.cc:686] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:37.572969 2372 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:37.630697 4367 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.600052 2578 raft_consensus.cc:2354] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3.
I20260806 03:51:37.600747 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.558959 4721 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:37.609946 2577 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:51:37.569093 4622 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.141s user 0.014s sys 0.010s
I20260806 03:51:37.612694 2376 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:37.614444 2332 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.531092 2377 raft_consensus.cc:595] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:37.629074 2534 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:37.599220 4748 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:37.574388 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.637225 4831 raft_consensus.cc:686] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:37.564939 4771 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260806 03:51:37.673971 4706 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.555802 2370 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:37.718153 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.609978 4276 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:37.660198 2480 raft_consensus.cc:2379] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:51:37.559182 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 301416us
I20260806 03:51:37.725217 2429 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.674211 2574 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:37.666186 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:37.678138 4833 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.661886 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.728556 2369 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:37.631618 4745 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:37.627980 3586 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.689591 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.510655 4472 raft_consensus.cc:686] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:37.660409 2479 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:37.564635 4830 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.682680 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:37.684765 4806 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:37.659776 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 117322us
I20260806 03:51:37.680248 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.765923 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:37.717783 2477 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:37.599543 2484 raft_consensus.cc:2354] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:37.696678 2377 raft_consensus.cc:603] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 03:51:37.697065 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 82771.1us
I20260806 03:51:37.759346 2375 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:37.659348 4600 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.658821 4539 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:37.634652 4702 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:37.702760 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1610) took 1269 ms. Trace:
I20260806 03:51:37.704365 2436 leader_election.cc:398] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3.
I20260806 03:51:37.789007 2436 leader_election.cc:401] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 03:51:37.788954 2372 rpcz_store.cc:276] 0806 03:51:36.433471 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.243796 (+810325us) service_pool.cc:227] Handling call
0806 03:51:37.702501 (+458705us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54000596d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.702677 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":434266880}
I20260806 03:51:37.550243 2481 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:51:37.790236 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.790380 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.790856 2372 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:37.705868 2376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1574) took 1934 ms (client timeout 2050 ms). Trace:
I20260806 03:51:37.707167 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 1981) took 1172 ms. Trace:
W20260806 03:51:37.791839 2376 rpcz_store.cc:269] 0806 03:51:35.771540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.135584 (+364044us) service_pool.cc:227] Handling call
0806 03:51:37.705653 (+1570069us) spinlock_profiling.cc:243] Waited 1.37 s on lock 0x7b5400059bd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.705794 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3157215616}
I20260806 03:51:37.716081 4412 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:37.633396 4792 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.719992 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.727322 2579 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:37.662079 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1611) took 1012 ms. Trace:
I20260806 03:51:37.809664 2374 rpcz_store.cc:276] 0806 03:51:36.649376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.496342 (+846966us) service_pool.cc:227] Handling call
0806 03:51:37.662015 (+165673us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20224}
I20260806 03:51:37.728528 3545 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:37.810792 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.757990 4581 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:37.736184 2370 raft_consensus.cc:1226] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 3. Ops: []
I20260806 03:51:37.816293 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 900) took 2502 ms. Trace:
I20260806 03:51:37.816749 2370 rpcz_store.cc:276] 0806 03:51:35.313647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.735362 (+2421715us) service_pool.cc:227] Handling call
0806 03:51:37.735894 (+ 532us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:37.815829 (+ 79935us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:37.816209 (+ 380us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.818629 2370 raft_consensus.cc:1226] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 4. Ops: []
I20260806 03:51:37.819375 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 901) took 2505 ms. Trace:
I20260806 03:51:37.747123 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.807296 4539 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.749292 2368 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.698580 4721 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:37.704917 4631 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:37.697888 4667 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.570125 3585 rpcz_store.cc:276] 0806 03:51:36.268424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.311745 (+ 43321us) service_pool.cc:227] Handling call
0806 03:51:37.412932 (+1101187us) spinlock_profiling.cc:243] Waited 802 ms on lock 0x7b54006ff9d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.424376 (+ 11444us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1844649856}
I20260806 03:51:37.685036 4772 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.766832 2373 raft_consensus.cc:2354] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:37.663609 2584 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:37.697860 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.773162 2332 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.776367 2331 leader_election.cc:398] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
I20260806 03:51:37.884799 2331 leader_election.cc:401] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260806 03:51:37.778795 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1911) took 1139 ms. Trace:
I20260806 03:51:37.733531 4528 raft_consensus.cc:2741] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 03:51:37.695497 4672 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.779163 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:44596 (request call id 13) took 2342 ms. Trace:
I20260806 03:51:37.790921 2481 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:37.674692 4766 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:37.793243 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:37.685616 4714 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.754194 4622 raft_consensus.cc:2741] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260806 03:51:37.757675 4790 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.792635 2577 rpcz_store.cc:276] 0806 03:51:36.535149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.535300 (+ 151us) service_pool.cc:227] Handling call
0806 03:51:37.706926 (+1171626us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540024c7d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.707091 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2327538560}
I20260806 03:51:37.678989 4722 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:37.811463 2374 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.924057 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.924806 2480 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:37.925683 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.926384 2480 raft_consensus.cc:2379] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:51:37.813927 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 85540us
W20260806 03:51:37.823199 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 194234us
I20260806 03:51:37.832533 2372 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:37.775640 4627 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.771543 4745 raft_consensus.cc:686] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:37.879170 2479 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:37.929531 2372 raft_consensus.cc:1226] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 3. Ops: []
I20260806 03:51:37.929886 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.885314 2484 rpcz_store.cc:276] 0806 03:51:36.639524 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.665664 (+ 26140us) service_pool.cc:227] Handling call
0806 03:51:37.778570 (+1112906us) spinlock_profiling.cc:243] Waited 783 ms on lock 0x7b54001216d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.778722 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1800362752}
W20260806 03:51:37.887933 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 114899us
I20260806 03:51:37.749426 4276 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.804214 4702 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:37.882787 2375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1587) took 2078 ms (client timeout 2462 ms). Trace:
W20260806 03:51:37.892252 4832 log.cc:924] Time spent T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.276s user 0.001s sys 0.000s
I20260806 03:51:37.812799 4730 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.891992 2377 rpcz_store.cc:276] 0806 03:51:35.430201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.530354 (+2100153us) service_pool.cc:227] Handling call
0806 03:51:37.772632 (+242278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.899471 2376 raft_consensus.cc:2354] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:37.740334 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1603) took 1668 ms. Trace:
I20260806 03:51:37.954087 2369 rpcz_store.cc:276] 0806 03:51:36.071434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.424632 (+1353198us) service_pool.cc:227] Handling call
0806 03:51:37.740204 (+315572us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.955300 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.819763 2370 rpcz_store.cc:276] 0806 03:51:35.313936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.817948 (+2504012us) service_pool.cc:227] Handling call
0806 03:51:37.818390 (+ 442us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:37.819091 (+ 701us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:37.819309 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.956120 2476 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:37.901309 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 125109us
I20260806 03:51:37.930311 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 902) took 2616 ms. Trace:
I20260806 03:51:37.956710 2372 rpcz_store.cc:276] 0806 03:51:35.314195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.929037 (+2614842us) service_pool.cc:227] Handling call
0806 03:51:37.929355 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:37.930055 (+ 700us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:37.930254 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.891119 4609 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:37.933231 2373 raft_consensus.cc:1226] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 3. Ops: []
W20260806 03:51:37.957827 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 77961.7us
I20260806 03:51:37.958314 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 904) took 2643 ms. Trace:
I20260806 03:51:37.933406 2332 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:37.931278 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.812680 3542 leader_election.cc:398] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:37.756228 4833 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.936585 2483 raft_consensus.cc:2354] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3.
I20260806 03:51:37.959771 2480 raft_consensus.cc:3037] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:37.958942 2481 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:37.961685 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 257487us
I20260806 03:51:37.958822 2373 rpcz_store.cc:276] 0806 03:51:35.314526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.932750 (+2618224us) service_pool.cc:227] Handling call
0806 03:51:37.933055 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:37.957963 (+ 24908us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:37.958235 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:37.904983 4584 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:37.937052 2534 leader_election.cc:398] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3.
I20260806 03:51:37.964830 2372 raft_consensus.cc:1226] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:37.964991 2534 leader_election.cc:401] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:51:37.905161 4412 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:37.888343 4771 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:37.805076 4600 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.922652 4722 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:37.962433 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.964279 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:37.906222 4744 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:37.966745 2370 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:37.967451 2369 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 5. Ops: []
I20260806 03:51:37.905754 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.878473 4581 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:37.878139 4728 consensus_queue.cc:237] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.049809 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:37.936678 2374 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:38.051308 2481 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:37.969178 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 117) took 2624 ms. Trace:
I20260806 03:51:37.969616 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 911) took 2640 ms. Trace:
I20260806 03:51:37.971115 2368 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:37.935499 4702 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:37.886515 4539 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.887238 4667 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.064888 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 910) took 2736 ms. Trace:
I20260806 03:51:37.967689 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:37.962306 4837 log.cc:924] Time spent T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.185s user 0.001s sys 0.000s
I20260806 03:51:37.940128 4790 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:38.014803 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 81478.9us
I20260806 03:51:38.009109 4664 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:38.044291 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 909) took 2728 ms. Trace:
I20260806 03:51:38.010349 4584 raft_consensus.cc:686] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:38.092170 2474 raft_consensus.cc:2379] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:38.093485 4539 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:38.004556 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 228900us
W20260806 03:51:37.934968 2375 rpcz_store.cc:269] 0806 03:51:35.800823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.045481 (+1244658us) service_pool.cc:227] Handling call
0806 03:51:37.871532 (+826051us) spinlock_profiling.cc:243] Waited 633 ms on lock 0x7b54003327d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:37.879174 (+ 7642us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1455212416}
I20260806 03:51:38.093690 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:38.094722 2474 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:38.094823 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:38.095513 2375 raft_consensus.cc:2362] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3. Re-sending same reply.
I20260806 03:51:38.007576 4771 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.057140 2374 raft_consensus.cc:1226] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 03:51:38.065351 2370 rpcz_store.cc:276] 0806 03:51:35.328494 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.966173 (+2637679us) service_pool.cc:227] Handling call
0806 03:51:37.966497 (+ 324us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.053326 (+ 86829us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.064810 (+ 11484us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.065887 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:38.066480 2479 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:38.014940 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 116) took 2671 ms. Trace:
I20260806 03:51:37.960619 4666 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:37.938509 4627 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:38.055150 4778 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.550s user 0.010s sys 0.004s
I20260806 03:51:37.994343 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1575) took 2221 ms. Trace:
W20260806 03:51:38.053325 3584 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.660s user 0.012s sys 0.000s
I20260806 03:51:38.053840 4722 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:38.063970 2476 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:38.124666 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:37.994906 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 66572.1us
I20260806 03:51:37.923108 4472 consensus_queue.cc:237] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:37.887760 4831 consensus_queue.cc:237] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.084034 2369 rpcz_store.cc:276] 0806 03:51:35.315324 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.963728 (+2648404us) service_pool.cc:227] Handling call
0806 03:51:37.967227 (+ 3499us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.043852 (+ 76625us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.044194 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.052400 2373 rpcz_store.cc:276] 0806 03:51:35.344367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.963036 (+2618669us) service_pool.cc:227] Handling call
0806 03:51:37.963397 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:37.968797 (+ 5400us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:37.968825 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:37.968869 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:37.968903 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:37.968986 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:37.969109 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.008849 4406 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3.
W20260806 03:51:38.101605 4688 consensus_meta.cc:287] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.666s user 0.007s sys 0.004s
I20260806 03:51:38.092888 4584 consensus_queue.cc:237] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.053592 2372 rpcz_store.cc:276] 0806 03:51:35.328706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.964287 (+2635581us) service_pool.cc:227] Handling call
0806 03:51:37.964616 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:37.969373 (+ 4757us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:37.969562 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.052829 2484 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:38.096532 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.116851 2481 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:38.085255 4772 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.117658 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1583) took 2330 ms. Trace:
I20260806 03:51:38.158893 2374 rpcz_store.cc:276] 0806 03:51:35.787400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.051648 (+2264248us) service_pool.cc:227] Handling call
0806 03:51:38.052054 (+ 406us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:38.117266 (+ 65212us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.117575 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.160255 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:38.161028 2374 raft_consensus.cc:2362] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2. Re-sending same reply.
I20260806 03:51:38.117763 2376 rpcz_store.cc:276] 0806 03:51:35.343990 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.961579 (+2617589us) service_pool.cc:227] Handling call
0806 03:51:38.014279 (+ 52700us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.014535 (+ 256us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:38.014572 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:38.014631 (+ 59us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:38.014674 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.014743 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:38.014859 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.069350 4609 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:38.119801 2377 rpcz_store.cc:276] 0806 03:51:35.772673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.993161 (+2220488us) service_pool.cc:227] Handling call
0806 03:51:37.993644 (+ 483us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:37.993898 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:37.993939 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:37.994005 (+ 66us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:37.994048 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:37.994117 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:37.994263 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.168771 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1607) took 1886 ms. Trace:
W20260806 03:51:38.079103 4807 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.642s user 0.012s sys 0.001s
I20260806 03:51:38.122057 4276 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.127286 2534 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:38.176178 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 48980.4us
I20260806 03:51:38.177192 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:38.096782 4840 log.cc:924] Time spent T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.209s user 0.000s sys 0.001s
I20260806 03:51:37.964223 4730 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:38.100857 4839 log.cc:924] Time spent T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.225s user 0.001s sys 0.000s
W20260806 03:51:38.101716 4841 log.cc:924] Time spent T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20260806 03:51:38.055832 4631 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3.
W20260806 03:51:38.100221 4842 log.cc:924] Time spent T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.208s user 0.002s sys 0.000s
I20260806 03:51:38.151692 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1598) took 2253 ms. Trace:
I20260806 03:51:38.117288 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.231652 2370 raft_consensus.cc:2354] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:38.123414 3584 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:51:38.157002 2375 raft_consensus.cc:2354] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3.
W20260806 03:51:38.185827 4662 consensus_meta.cc:287] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.650s user 0.015s sys 0.000s
I20260806 03:51:38.269307 2577 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:38.269536 4662 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.151654 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.272579 2373 raft_consensus.cc:2354] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:38.274070 2373 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:38.274933 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 128) took 1675 ms. Trace:
I20260806 03:51:38.157404 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:38.113886 4664 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:38.280211 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 50011us
I20260806 03:51:38.282346 4664 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.164886 4830 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.094745 4539 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.284003 4539 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.095932 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:38.132100 4836 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.430s user 0.000s sys 0.002s
I20260806 03:51:38.169180 2377 rpcz_store.cc:276] 0806 03:51:36.282880 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.167737 (+1884857us) service_pool.cc:227] Handling call
0806 03:51:38.168140 (+ 403us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.168430 (+ 290us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:38.168463 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:38.168502 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:38.168536 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.168591 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:38.168710 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.169579 2480 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:38.170712 2481 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:38.294566 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:38.295624 2481 raft_consensus.cc:2379] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:38.174939 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:38.180496 4600 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.182507 2534 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:38.313530 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 131088us
I20260806 03:51:38.315183 2534 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:38.182641 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.148339 4806 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.185257 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1600) took 2228 ms. Trace:
I20260806 03:51:38.332696 2376 rpcz_store.cc:276] 0806 03:51:35.956776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.163094 (+2206318us) service_pool.cc:227] Handling call
0806 03:51:38.184600 (+ 21506us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.184838 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:38.184878 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:38.184936 (+ 58us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:38.184979 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.185047 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:38.185182 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.233620 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:38.114181 4716 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.231148 2372 rpcz_store.cc:276] 0806 03:51:35.898701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.150793 (+2252092us) service_pool.cc:227] Handling call
0806 03:51:38.151181 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.151391 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:38.151417 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:38.151458 (+ 41us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:38.151487 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.151534 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:38.151636 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.340062 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.341204 2372 raft_consensus.cc:2379] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:38.334489 2375 raft_consensus.cc:2362] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2. Re-sending same reply.
I20260806 03:51:38.227627 4743 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:38.166328 4797 raft_consensus.cc:2932] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:38.239722 4833 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.113828 4745 consensus_queue.cc:237] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.242431 4730 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.224622 4276 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.393321 4276 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.190943 4848 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:38.396500 4848 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:38.396554 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.397468 4848 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:38.398033 2369 raft_consensus.cc:2379] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:38.399300 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.242978 4672 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.245033 2370 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:38.183543 4853 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.400870 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1609) took 1981 ms. Trace:
I20260806 03:51:38.401365 2370 rpcz_store.cc:276] 0806 03:51:36.419854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.240835 (+1820981us) service_pool.cc:227] Handling call
0806 03:51:38.244505 (+ 3670us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.400494 (+155989us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.400792 (+ 298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.400996 4853 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.402392 4743 raft_consensus.cc:686] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:38.402716 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.156646 4688 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.404373 2583 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:38.403570 2370 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:38.252238 2477 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:51:38.248054 4412 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3.
I20260806 03:51:38.259068 4771 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.261128 4790 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.175125 4807 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.239610 4714 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.422552 4807 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.423884 2482 raft_consensus.cc:2379] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:38.268632 4792 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.290817 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:38.387179 3586 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.621s user 0.012s sys 0.000s
I20260806 03:51:38.275357 2373 rpcz_store.cc:276] 0806 03:51:36.599185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.273294 (+1674109us) service_pool.cc:227] Handling call
0806 03:51:38.273775 (+ 481us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.274559 (+ 784us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.274828 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.297874 2481 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:51:38.438987 2481 raft_consensus.cc:595] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:38.439461 2481 raft_consensus.cc:603] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:51:38.439137 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.317868 2534 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
W20260806 03:51:38.440899 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 123131us
I20260806 03:51:38.441854 2534 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:38.326253 4539 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.287498 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:38.435468 4772 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.444877 2474 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:38.330149 2484 raft_consensus.cc:1226] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 03:51:38.385490 4716 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.342216 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.452864 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.342790 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:38.385883 2479 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:38.462081 2375 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:38.408818 2483 raft_consensus.cc:1261] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:38.410604 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.463771 3547 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:38.464399 2370 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:38.426440 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:38.465534 2376 raft_consensus.cc:2354] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:38.427824 2332 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:38.467108 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:38.430855 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.433328 2480 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:38.459520 2481 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:38.290349 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:38.469574 2377 raft_consensus.cc:2354] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:38.469803 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:38.471148 2377 raft_consensus.cc:1261] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:38.336633 4675 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:38.460242 2372 raft_consensus.cc:2354] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5.
I20260806 03:51:38.471894 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1613) took 1581 ms. Trace:
I20260806 03:51:38.480172 2377 rpcz_store.cc:276] 0806 03:51:36.890523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.470279 (+1579756us) service_pool.cc:227] Handling call
0806 03:51:38.470691 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.471591 (+ 900us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.471828 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.436816 2476 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:38.354779 4664 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:38.482275 2377 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:38.484086 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:38.485160 2370 raft_consensus.cc:2354] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3.
I20260806 03:51:38.486107 2479 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:38.488226 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.489404 4675 raft_consensus.cc:686] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:38.489786 2372 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:38.495309 2332 leader_election.cc:398] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:38.496840 2429 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:38.504976 4706 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.757s user 0.016s sys 0.000s
I20260806 03:51:38.505067 4667 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.505689 4706 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:38.507370 2379 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:38.491820 4618 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:38.508385 4664 raft_consensus.cc:686] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:38.444903 4772 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.510135 4664 consensus_queue.cc:237] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.491662 4778 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:38.404202 4743 consensus_queue.cc:237] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.493407 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:38.519631 2375 raft_consensus.cc:2379] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:38.520952 4688 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:38.492108 4539 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.530311 2376 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:38.518222 4600 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:38.532867 4600 raft_consensus.cc:686] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:38.442996 3586 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:38.477499 4730 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:38.542845 2369 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:38.492368 4375 raft_consensus.cc:2781] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:38.548767 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:38.549710 2480 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:38.544464 2481 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:38.551662 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1617) took 1137 ms. Trace:
I20260806 03:51:38.492626 4622 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.501163 4714 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:38.512887 4778 raft_consensus.cc:686] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:38.518504 4792 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:38.514613 4716 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.569664 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 1993) took 1536 ms. Trace:
I20260806 03:51:38.553547 2479 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 03:51:38.526588 4583 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260806 03:51:38.583088 2583 rpcz_store.cc:276] 0806 03:51:37.033219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.033597 (+ 378us) service_pool.cc:227] Handling call
0806 03:51:38.569359 (+1535762us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400238554. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:38.569579 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3125239552}
I20260806 03:51:38.583992 2479 pending_rounds.cc:76] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 1
I20260806 03:51:38.565882 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:38.552203 2369 rpcz_store.cc:276] 0806 03:51:37.414069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.494097 (+1080028us) service_pool.cc:227] Handling call
0806 03:51:38.542434 (+ 48337us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.551294 (+ 8860us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:38.551583 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.600870 2480 raft_consensus.cc:2932] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:38.602656 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:38.603817 2481 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:38.491302 4722 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.511950 4772 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.518405 4736 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260806 03:51:38.556862 4730 raft_consensus.cc:686] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:38.534565 4600 consensus_queue.cc:237] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.546562 3586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 316) took 1153 ms. Trace:
I20260806 03:51:38.596835 2479 pending_rounds.cc:97] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 03:51:38.692504 2479 raft_consensus.cc:2866] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:38.613552 4778 consensus_queue.cc:237] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.575887 4622 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.491353 4675 consensus_queue.cc:237] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.572705 4375 raft_consensus.cc:686] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:38.562809 4714 raft_consensus.cc:686] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:38.536226 4539 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.692312 3586 rpcz_store.cc:276] 0806 03:51:37.392814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.398131 (+ 5317us) service_pool.cc:227] Handling call
0806 03:51:38.545286 (+1147155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:38.562323 2474 raft_consensus.cc:1261] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:38.563499 4792 raft_consensus.cc:686] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:38.624110 2477 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:38.625711 4662 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.610170 2481 raft_consensus.cc:1261] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:38.655588 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1925) took 1108 ms. Trace:
I20260806 03:51:38.750692 2482 rpcz_store.cc:276] 0806 03:51:37.547604 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.547733 (+ 129us) service_pool.cc:227] Handling call
0806 03:51:37.600516 (+ 52783us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:51:38.655320 (+1054804us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b540019f354. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:38.655507 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1682665088}
I20260806 03:51:38.658053 3592 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:38.662150 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1622) took 1040 ms. Trace:
I20260806 03:51:38.753163 2379 rpcz_store.cc:276] 0806 03:51:37.622071 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.661576 (+ 39505us) service_pool.cc:227] Handling call
0806 03:51:38.661898 (+1000322us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b5400012cd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:38.662072 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1768232704}
I20260806 03:51:38.649991 2583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:51:38.647059 2538 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:38.753026 4792 consensus_queue.cc:237] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.755017 2583 raft_consensus.cc:595] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
W20260806 03:51:38.755748 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 108864us
I20260806 03:51:38.756705 2538 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:38.756973 4662 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:38.758011 4662 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:38.759148 2538 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:38.699121 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 51105.9us
I20260806 03:51:38.701716 2375 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:38.760305 2429 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:38.761870 2429 leader_election.cc:398] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3.
I20260806 03:51:38.761682 4375 consensus_queue.cc:237] T ac682b8be46c4b6599610372a4c8c727 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.762969 2429 leader_election.cc:398] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:38.764380 4807 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:38.677066 4722 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.765350 4807 raft_consensus.cc:686] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:38.768889 4734 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:38.768191 4714 consensus_queue.cc:237] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.670297 4276 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.763937 2429 leader_election.cc:398] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:38.775071 2429 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:38.736850 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1928) took 1187 ms. Trace:
I20260806 03:51:38.738863 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:38.778337 2369 raft_consensus.cc:2379] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:38.746973 4539 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.672497 4627 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.688135 4791 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.538782 4702 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.694027 4706 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.773013 2538 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:38.799954 2429 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:38.775870 2478 rpcz_store.cc:276] 0806 03:51:37.549405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.559563 (+ 10158us) service_pool.cc:227] Handling call
0806 03:51:37.560102 (+ 539us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:37.560573 (+ 471us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:37.560615 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:37.565944 (+ 5329us) log.cc:842] Serialized 26 byte log entry
0806 03:51:37.877090 (+311146us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:37.877135 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:37.877210 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:38.567743 (+690533us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b24001373d8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d799f5cf4 00007f0d799f3dfb 00007f0d79a82559 00007f0d79a821e5 00007f0d79a81740 00007f0d7c774578 00007f0d7c774cd8 00007f0d7c74c6e8 00007f0d7c747aea 00007f0d7c746f36
0806 03:51:38.715836 (+148093us) raft_consensus.cc:1671] finished
0806 03:51:38.715879 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:38.736719 (+ 20840us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":100,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":8282,"spinlock_wait_cycles":34742528,"thread_start_us":215401,"threads_started":1,"wal-append.queue_time_us":281376}
I20260806 03:51:38.784055 4853 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.800763 2538 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:38.788437 4791 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:38.791481 4367 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.072s user 0.015s sys 0.000s
I20260806 03:51:38.794098 4622 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.802109 2429 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:38.803949 2538 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:38.805210 4688 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:38.805210 4276 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:38.841795 4688 raft_consensus.cc:686] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:38.842521 4766 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:38.805977 4748 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:51:38.674321 4730 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.820713 4581 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:38.826766 4702 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.836735 4651 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:38.848235 4791 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:38.848793 4651 raft_consensus.cc:686] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:38.838542 2538 leader_election.cc:398] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3.
I20260806 03:51:38.840803 2429 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:38.841857 4276 raft_consensus.cc:686] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:38.767302 4807 consensus_queue.cc:237] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.854696 2429 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:38.856338 2429 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:51:38.834491 4721 consensus_meta.cc:287] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.932s user 0.007s sys 0.006s
I20260806 03:51:38.868968 2429 leader_election.cc:398] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:38.869563 2429 leader_election.cc:401] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260806 03:51:38.870985 2429 leader_election.cc:398] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5.
I20260806 03:51:38.871348 2429 leader_election.cc:401] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:51:38.828083 4662 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:38.874861 2429 leader_election.cc:398] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:38.875672 4651 consensus_queue.cc:237] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:38.862885 4581 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:38.878575 4528 consensus_queue.cc:1035] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:38.843344 4688 consensus_queue.cc:237] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:38.880110 4766 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:38.833725 4367 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:38.973412 4715 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.981s user 0.018s sys 0.041s
W20260806 03:51:39.012645 4584 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.825s user 0.000s sys 0.006s
W20260806 03:51:38.980098 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.023s user 0.029s sys 0.109s
W20260806 03:51:39.041105 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.084s user 0.029s sys 0.109s
W20260806 03:51:39.041615 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 1.08455e+06us
I20260806 03:51:38.880782 4853 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:51:38.973599 4838 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.916s user 0.006s sys 0.052s
I20260806 03:51:38.971890 4702 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:38.873736 4721 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:38.879925 4790 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5.
W20260806 03:51:38.997138 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 123497us
W20260806 03:51:39.002440 2584 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.930s user 0.014s sys 0.062s
W20260806 03:51:39.054390 2584 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.982s user 0.015s sys 0.062s
W20260806 03:51:39.007133 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 132453us
I20260806 03:51:39.054752 2538 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:38.886716 4406 consensus_queue.cc:1046] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20260806 03:51:39.060817 2574 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3. Ops: []
W20260806 03:51:39.037834 4715 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.069s user 0.019s sys 0.041s
I20260806 03:51:38.865293 4622 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:51:39.039994 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.964s user 0.029s sys 0.063s
W20260806 03:51:39.102943 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.027s user 0.031s sys 0.065s
I20260806 03:51:39.011983 4581 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:39.040901 4584 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s user 0.000s sys 0.006s
W20260806 03:51:39.048054 4838 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.997s user 0.006s sys 0.053s
I20260806 03:51:39.049400 2331 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:39.105630 3591 raft_consensus.cc:2354] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:39.048456 4853 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:39.116499 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 67208us
I20260806 03:51:38.987753 4276 consensus_queue.cc:237] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:39.061836 4406 raft_consensus.cc:3032] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:39.104843 4581 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:39.106204 4528 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:39.142174 4528 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:39.139127 4748 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 03:51:39.145612 3542 leader_election.cc:398] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:39.146066 3542 leader_election.cc:401] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:51:39.147292 4584 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
I20260806 03:51:39.109720 4622 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.142688 4528 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:39.061993 4702 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.124410 4715 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:39.122093 4853 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:39.140130 4406 raft_consensus.cc:729] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:39.149493 4797 consensus_meta.cc:287] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.723s user 0.018s sys 0.000s
I20260806 03:51:39.177876 3542 leader_election.cc:398] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3.
W20260806 03:51:39.149833 4456 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.383s user 0.013s sys 0.042s
W20260806 03:51:39.203001 2329 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.030s user 0.017s sys 0.044s
W20260806 03:51:39.203434 2329 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.237s user 0.017s sys 0.044s
I20260806 03:51:39.205283 4721 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:39.206557 4406 consensus_queue.cc:260] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:39.208148 4406 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:39.214362 4766 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4.
I20260806 03:51:39.216246 4715 raft_consensus.cc:686] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:39.218868 4367 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:39.150244 4584 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:39.220430 3545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.073s user 0.021s sys 0.045s
W20260806 03:51:39.220786 3545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.097s user 0.021s sys 0.045s
I20260806 03:51:39.220577 4584 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.222872 4584 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:39.123164 4790 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.076s
I20260806 03:51:39.223971 4584 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.224459 4584 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.226737 4584 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:39.165112 4772 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:39.198267 4860 log.cc:924] Time spent T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260806 03:51:39.232815 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 921) took 1236 ms. Trace:
I20260806 03:51:39.234776 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:51:39.163182 4855 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.057s user 0.000s sys 0.021s
I20260806 03:51:39.219672 4367 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:39.263360 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:39.264716 2373 raft_consensus.cc:1261] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:39.269858 2369 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:39.270321 4627 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.270911 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:39.274112 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2005) took 1510 ms. Trace:
W20260806 03:51:39.274650 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.193s user 0.016s sys 0.041s
I20260806 03:51:39.239203 2584 rpcz_store.cc:276] 0806 03:51:37.996665 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.996816 (+ 151us) service_pool.cc:227] Handling call
0806 03:51:37.997241 (+ 425us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:38.005308 (+ 8067us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:38.005354 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:38.049549 (+ 44195us) log.cc:842] Serialized 26 byte log entry
0806 03:51:39.054887 (+1005338us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:39.054950 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:39.055023 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:39.232460 (+177437us) raft_consensus.cc:1671] finished
0806 03:51:39.232499 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:39.232724 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8726,"raft.queue_time_us":8299,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":63636,"thread_start_us":982616,"threads_started":1,"wal-append.queue_time_us":963997}
I20260806 03:51:39.279063 2372 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260806 03:51:39.284922 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.204s user 0.016s sys 0.041s
I20260806 03:51:39.284759 2574 rpcz_store.cc:276] 0806 03:51:37.763988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.775514 (+ 11526us) service_pool.cc:227] Handling call
0806 03:51:37.775887 (+ 373us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:39.061375 (+1285488us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:39.273736 (+212361us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b540013c4d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:39.274030 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2938228480}
I20260806 03:51:39.271611 4722 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.278174 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:39.197373 4736 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:39.249245 4456 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.486s user 0.014s sys 0.043s
I20260806 03:51:39.293025 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:39.277287 4855 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.175s user 0.000s sys 0.022s
I20260806 03:51:39.315181 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:39.227950 4702 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:39.331769 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:39.346174 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:39.347076 2478 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:39.343523 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:39.342810 2374 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.008s user 0.005s sys 0.027s
W20260806 03:51:39.347854 2374 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.013s user 0.005s sys 0.027s
I20260806 03:51:39.348654 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:39.351707 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:39.357672 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:39.358668 2478 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:39.293010 4862 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:39.333751 4697 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.378s user 0.016s sys 0.018s
W20260806 03:51:39.293087 4831 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.868s user 0.007s sys 0.023s
I20260806 03:51:39.165578 4622 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:39.381619 4622 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:39.383059 4831 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.958s user 0.008s sys 0.023s
W20260806 03:51:39.307310 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 129633us
I20260806 03:51:39.367620 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:39.308357 4627 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.385936 3585 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:39.389941 4584 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.390600 4584 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.391273 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:39.361728 4854 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:39.333083 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:39.392391 4854 raft_consensus.cc:686] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:39.377074 2379 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:39.229300 4772 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:39.308465 4604 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.152s user 0.004s sys 0.017s
I20260806 03:51:39.399523 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:39.344275 4856 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:39.394279 4697 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.444s user 0.016s sys 0.019s
I20260806 03:51:39.272504 4715 consensus_queue.cc:237] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:39.387840 4627 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:39.372951 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:39.303174 4367 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.316531 4722 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.394368 4862 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:39.406335 4604 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.292s user 0.005s sys 0.017s
I20260806 03:51:39.451560 2481 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:39.472524 2375 raft_consensus.cc:2379] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:39.455338 2332 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:39.475767 2332 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:51:39.409067 4472 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.887s user 0.011s sys 0.026s
I20260806 03:51:39.507463 4627 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:39.508316 4627 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:39.508471 4873 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260806 03:51:39.508829 4627 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:39.451025 4856 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:39.511426 4367 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:39.513634 4855 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:39.514324 4855 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:39.516921 4857 log.cc:924] Time spent T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.343s user 0.000s sys 0.001s
I20260806 03:51:39.511438 4856 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:39.522403 4853 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:39.468525 4702 consensus_queue.cc:1046] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260806 03:51:39.532534 4584 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:39.344765 4736 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:39.537654 4472 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.016s user 0.012s sys 0.027s
I20260806 03:51:39.534188 4528 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.452474 4868 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:39.535652 4791 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:39.534852 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:39.564486 4706 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.586294 4662 raft_consensus.cc:979] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:39.393491 4854 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.639520 3586 raft_consensus.cc:2379] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:39.640841 4855 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:39.630587 4791 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:39.626374 4766 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:39.638862 4848 consensus_meta.cc:287] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.241s user 0.006s sys 0.014s
W20260806 03:51:39.631270 4600 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.012s sys 0.026s
I20260806 03:51:39.645809 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:39.638366 4706 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.687817 4706 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:39.645031 4855 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.641387 4831 raft_consensus.cc:1067] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:51:39.690124 4706 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:39.690876 4600 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.858s user 0.012s sys 0.026s
W20260806 03:51:39.697948 2480 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.022s sys 0.015s
I20260806 03:51:39.647311 4791 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:39.700129 2480 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s user 0.023s sys 0.015s
I20260806 03:51:39.701150 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1645) took 1114 ms. Trace:
I20260806 03:51:39.701587 2480 rpcz_store.cc:276] 0806 03:51:38.586615 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.586829 (+ 214us) service_pool.cc:227] Handling call
0806 03:51:38.600500 (+ 13671us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.600738 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:51:39.003603 (+402865us) log.cc:842] Serialized 14 byte log entry
0806 03:51:39.700660 (+697057us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:39.700739 (+ 79us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:39.700802 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:39.700896 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:39.701071 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":696587,"threads_started":1}
W20260806 03:51:39.703118 2368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.575s user 0.002s sys 0.027s
W20260806 03:51:39.695003 4664 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.018s sys 0.025s
I20260806 03:51:39.632282 4868 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:39.643241 4627 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:39.703517 2368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.575s user 0.002s sys 0.027s
I20260806 03:51:39.719806 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:39.650789 4622 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
W20260806 03:51:39.788508 3591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36484 (request call id 258) took 2297 ms (client timeout 1569 ms). Trace:
I20260806 03:51:39.795840 2482 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:51:39.708217 4664 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s user 0.018s sys 0.025s
W20260806 03:51:39.794914 3591 rpcz_store.cc:269] 0806 03:51:37.414319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:37.414482 (+ 163us) service_pool.cc:227] Handling call
0806 03:51:39.711224 (+2296742us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400531454. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:39.711398 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3636396800}
W20260806 03:51:39.770972 4621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.546s user 0.001s sys 0.054s
I20260806 03:51:39.690474 4848 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:39.709746 4666 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.587s user 0.042s sys 0.117s
W20260806 03:51:39.800318 2483 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.253s user 0.023s sys 0.029s
I20260806 03:51:39.693732 4855 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:39.866688 2483 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.320s user 0.024s sys 0.029s
I20260806 03:51:39.709064 4472 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:39.690285 4766 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:39.798199 4772 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:39.698828 4706 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:39.862767 4621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.671s user 0.002s sys 0.055s
W20260806 03:51:39.800238 4859 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s user 0.015s sys 0.035s
I20260806 03:51:39.862882 4600 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:39.873380 4666 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.751s user 0.042s sys 0.117s
I20260806 03:51:39.862900 4853 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:39.924008 2476 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.000s sys 0.027s
W20260806 03:51:39.881680 4745 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.334s user 0.029s sys 0.099s
W20260806 03:51:39.884539 4581 consensus_meta.cc:287] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.738s user 0.012s sys 0.002s
W20260806 03:51:39.924672 4743 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.407s user 0.017s sys 0.048s
W20260806 03:51:39.931483 2534 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.224s user 0.027s sys 0.023s
W20260806 03:51:39.983702 2534 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.293s user 0.027s sys 0.023s
W20260806 03:51:39.984472 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 1.29427e+06us
W20260806 03:51:39.935827 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.805s user 0.018s sys 0.005s
W20260806 03:51:39.961292 4745 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.402s user 0.030s sys 0.100s
W20260806 03:51:39.934602 2476 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.617s user 0.000s sys 0.027s
W20260806 03:51:39.967366 2429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.911s user 0.046s sys 0.071s
W20260806 03:51:40.011363 2429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.955s user 0.048s sys 0.071s
W20260806 03:51:40.011864 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 956142us
I20260806 03:51:40.013041 2429 leader_election.cc:398] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3.
I20260806 03:51:40.015439 2429 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:39.873042 4865 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:39.925340 4859 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.748s user 0.016s sys 0.035s
W20260806 03:51:39.911156 4714 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s user 0.011s sys 0.012s
W20260806 03:51:39.989537 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.861s user 0.019s sys 0.005s
W20260806 03:51:40.068064 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 951498us
W20260806 03:51:39.800093 4730 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.019s sys 0.032s
I20260806 03:51:39.933254 4722 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:39.934689 4621 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:39.983314 4743 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.466s user 0.017s sys 0.049s
I20260806 03:51:39.936297 4528 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:39.987919 4728 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.304s user 0.059s sys 0.093s
I20260806 03:51:39.904771 4472 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:40.027273 4714 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.988s user 0.011s sys 0.012s
I20260806 03:51:40.008697 4664 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.982859 4581 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.393594 4766 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.394572 4766 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:40.349929 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.811s user 0.012s sys 0.002s
I20260806 03:51:40.395509 4766 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:40.396368 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.858s user 0.013s sys 0.002s
I20260806 03:51:40.398496 4766 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.399163 4766 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.352468 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1592) took 4549 ms. Trace:
I20260806 03:51:40.399995 4766 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.400283 2368 rpcz_store.cc:276] 0806 03:51:35.802726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.054216 (+2251490us) service_pool.cc:227] Handling call
0806 03:51:38.055239 (+ 1023us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:38.055753 (+ 514us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:38.055790 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:38.080616 (+ 24826us) log.cc:842] Serialized 548 byte log entry
0806 03:51:39.787919 (+1707303us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:39.787970 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:39.788088 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:40.351474 (+563386us) raft_consensus.cc:1671] finished
0806 03:51:40.351554 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:40.352116 (+ 562us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1659236,"threads_started":1,"wal-append.queue_time_us":2122451}
I20260806 03:51:40.402801 4766 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:40.353819 2329 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.846s user 0.009s sys 0.001s
I20260806 03:51:40.403401 4766 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:40.403698 2329 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.896s user 0.009s sys 0.001s
W20260806 03:51:40.354527 2433 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.788s user 0.000s sys 0.015s
W20260806 03:51:40.406772 2433 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.840s user 0.001s sys 0.016s
I20260806 03:51:40.407119 4790 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:40.072341 4778 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.221s user 0.001s sys 0.022s
W20260806 03:51:40.408643 4778 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.572s user 0.002s sys 0.023s
I20260806 03:51:40.408460 4790 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:40.069491 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.025s sys 0.014s
W20260806 03:51:40.412509 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.848s user 0.025s sys 0.014s
W20260806 03:51:40.413010 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 955829us
W20260806 03:51:40.360073 2438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.776s user 0.006s sys 0.005s
W20260806 03:51:40.415916 2438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.851s user 0.007s sys 0.005s
W20260806 03:51:40.011870 2484 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.562s user 0.004s sys 0.030s
W20260806 03:51:40.361187 2537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.235s user 0.032s sys 0.078s
I20260806 03:51:40.417208 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:40.417871 2537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.292s user 0.033s sys 0.078s
W20260806 03:51:40.417955 2484 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.968s user 0.004s sys 0.030s
W20260806 03:51:40.368848 3545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.927s user 0.007s sys 0.020s
W20260806 03:51:40.418778 3545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.977s user 0.007s sys 0.020s
I20260806 03:51:40.071653 2538 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:40.421438 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 349756us
I20260806 03:51:40.423806 4662 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:40.424620 4662 raft_consensus.cc:686] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:40.162791 4792 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.867s user 0.015s sys 0.018s
I20260806 03:51:40.378803 3592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36484 (request call id 262) took 1014 ms. Trace:
W20260806 03:51:40.380347 4276 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.154s user 0.011s sys 0.020s
W20260806 03:51:40.039542 4375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.877s user 0.005s sys 0.012s
W20260806 03:51:40.381072 4584 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.010s sys 0.016s
I20260806 03:51:40.068974 4367 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:40.381778 4651 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.236s user 0.017s sys 0.022s
I20260806 03:51:40.383005 4456 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:39.937444 4600 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:40.172176 4528 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.188535 4621 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:40.388331 4734 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.619s user 0.015s sys 0.010s
I20260806 03:51:40.174685 4736 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.530102 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:40.171945 4675 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.314s user 0.009s sys 0.027s
I20260806 03:51:40.531680 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.531840 4748 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:40.532572 4675 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.680s user 0.009s sys 0.030s
I20260806 03:51:40.533062 4748 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.533779 4675 consensus_queue.cc:1035] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260806 03:51:40.072557 4688 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.011s sys 0.013s
W20260806 03:51:40.535241 4688 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.325s user 0.016s sys 0.020s
I20260806 03:51:40.519328 4662 consensus_queue.cc:237] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.529189 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.537631 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:40.071763 4730 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.099s user 0.020s sys 0.032s
I20260806 03:51:40.071935 4745 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:40.168146 4728 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.485s user 0.060s sys 0.101s
I20260806 03:51:40.392629 4706 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.387421 4627 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.671600 4627 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.672478 4627 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.411713 4790 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:40.419518 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 1.00283e+06us
W20260806 03:51:40.369081 3542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.906s user 0.018s sys 0.038s
I20260806 03:51:40.066686 4853 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:40.676433 3542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.214s user 0.018s sys 0.038s
I20260806 03:51:40.676151 4675 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:40.257807 4868 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:40.278677 4869 log.cc:924] Time spent T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.290s user 0.000s sys 0.001s
W20260806 03:51:40.297307 4870 log.cc:924] Time spent T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260806 03:51:40.679749 4923 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.110s user 0.004s sys 0.001s
W20260806 03:51:40.283747 4872 log.cc:924] Time spent T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.300s user 0.000s sys 0.001s
W20260806 03:51:40.279616 4874 log.cc:924] Time spent T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
W20260806 03:51:40.290426 4875 log.cc:924] Time spent T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.113s user 0.000s sys 0.001s
I20260806 03:51:40.687922 4745 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:40.689035 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:40.280311 4876 log.cc:924] Time spent T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.294s user 0.000s sys 0.001s
W20260806 03:51:40.284612 4884 log.cc:924] Time spent T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.060s user 0.000s sys 0.001s
I20260806 03:51:40.691550 4862 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:40.310722 4887 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 03:51:40.267352 4890 consensus_queue.cc:1046] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260806 03:51:40.694289 4890 raft_consensus.cc:3032] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:40.694653 4890 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:40.372960 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:40.683262 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:40.483433 3592 rpcz_store.cc:276] 0806 03:51:39.363750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:39.364057 (+ 307us) service_pool.cc:227] Handling call
0806 03:51:39.710282 (+346225us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b5800000f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:39.710609 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:39.813980 (+103371us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:39.814028 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:39.814097 (+ 69us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:39.814142 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:40.377989 (+563847us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54005fe954. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:40.378077 (+ 88us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:40.378474 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245284864}
I20260806 03:51:40.696740 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:40.443279 4714 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:51:40.506559 4584 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.972s user 0.010s sys 0.016s
W20260806 03:51:40.506785 4276 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.281s user 0.012s sys 0.021s
W20260806 03:51:40.510998 4651 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.364s user 0.018s sys 0.022s
W20260806 03:51:40.524595 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 456847us
I20260806 03:51:40.706552 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1599) took 4779 ms. Trace:
I20260806 03:51:40.665959 4715 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.707077 2374 rpcz_store.cc:276] 0806 03:51:35.931227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.161843 (+2230616us) service_pool.cc:227] Handling call
0806 03:51:38.162310 (+ 467us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:38.163308 (+ 998us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:38.163349 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:38.177564 (+ 14215us) log.cc:842] Serialized 26 byte log entry
0806 03:51:39.348285 (+1170721us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:39.348329 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:39.348414 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:40.685710 (+1337296us) raft_consensus.cc:1671] finished
0806 03:51:40.702217 (+ 16507us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:40.706289 (+ 4072us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13854,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":43691,"thread_start_us":1018858,"threads_started":1,"wal-append.queue_time_us":1638357}
I20260806 03:51:40.530167 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.708622 4715 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.710961 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.533931 4894 consensus_queue.cc:1035] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:40.534426 4600 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.219605 4664 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:40.389576 4697 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.924s user 0.022s sys 0.052s
I20260806 03:51:40.256476 4848 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:40.715401 4697 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.250s user 0.022s sys 0.052s
I20260806 03:51:40.260605 4855 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.538682 2583 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:40.685343 3586 raft_consensus.cc:2932] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:40.711740 4715 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.511615 4766 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.553808 2232 catalog_manager.cc:5495] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:40.556746 4906 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:40.557396 4581 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.557746 4743 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:40.558334 4871 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:40.558528 4909 raft_consensus.cc:3032] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:40.558185 4367 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.732364 2478 raft_consensus.cc:1261] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:40.733048 4743 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.732472 4871 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.735225 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:40.558821 4915 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.539533 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:40.559285 2483 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s user 0.004s sys 0.013s
I20260806 03:51:40.557240 4910 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:40.587608 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58951.8us
I20260806 03:51:40.552650 4734 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:40.557550 4472 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.681217 4539 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:40.583271 4797 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.251s user 0.006s sys 0.045s
I20260806 03:51:40.745039 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:40.745427 4797 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.517s user 0.006s sys 0.046s
I20260806 03:51:40.602176 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.593734 4748 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:40.612459 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1637) took 2509 ms. Trace:
I20260806 03:51:40.591625 4456 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.748962 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:40.750046 2480 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:40.622973 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:40.611238 4807 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.428s user 0.028s sys 0.016s
I20260806 03:51:40.636077 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:40.630252 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.988s user 0.004s sys 0.011s
W20260806 03:51:40.590608 4854 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.053s sys 0.106s
W20260806 03:51:40.602442 2476 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s user 0.002s sys 0.011s
I20260806 03:51:40.560050 4831 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260806 03:51:40.635421 4772 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.611127 4920 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:40.611148 4666 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:40.600816 4722 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:40.628654 4615 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.797282 4722 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.797452 4615 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.657814 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.661799 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:40.801429 2476 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s user 0.002s sys 0.015s
I20260806 03:51:40.802676 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:40.802822 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:40.801683 2537 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.572s (SENT) [suppressed 6 similar messages]
I20260806 03:51:40.664649 2576 raft_consensus.cc:1261] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:40.665274 4728 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:40.804147 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:40.787881 4807 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.666s user 0.029s sys 0.016s
I20260806 03:51:40.665926 4913 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.611618 4721 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.669829 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.067433 4865 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.821445 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.821008 4913 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.822327 2576 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:40.823457 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.823618 4865 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:40.676997 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 1.26832e+06us
I20260806 03:51:40.678925 4621 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.682744 3591 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:40.687466 2482 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:40.687383 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:40.692756 4926 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.675732 2236 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38220 (request call id 126) took 5515 ms. Trace:
I20260806 03:51:40.696760 3585 raft_consensus.cc:2379] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:40.703570 4714 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:40.529098 4375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.357s user 0.016s sys 0.038s
I20260806 03:51:40.714718 4664 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:40.831698 4721 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.717077 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.831954 4772 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.832705 4714 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:40.833110 2583 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:40.834306 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.834462 4721 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:40.834597 4772 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.835062 2583 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:40.539592 2368 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:40.711788 2584 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:40.836804 2376 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:40.837932 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:40.838667 2584 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 4
I20260806 03:51:40.838541 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:40.839391 2376 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:40.732121 4909 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:40.841179 4909 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.842453 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:40.750263 4456 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:40.826732 2236 rpcz_store.cc:276] 0806 03:51:35.159862 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:35.173269 (+ 13407us) service_pool.cc:227] Handling call
0806 03:51:35.254812 (+ 81543us) spinlock_profiling.cc:243] Waited 38.9 ms on lock 0x7b54000e3154. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4aaf0 00007f0d7cacdf2a 00007f0d82a1fd6c 00007f0d82b19c3e 00007f0d80e2bfb4 00007f0d80e2bf42 00007f0d80e2be71 00007f0d80e2bdec 00007f0d80e2b0a2 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:40.675374 (+5420562us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:36.744246 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:36.989547 (+245301us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:36.995017 (+ 5470us) write_op.cc:435] Acquired schema lock
0806 03:51:36.999233 (+ 4216us) tablet.cc:660] Decoding operations
0806 03:51:37.088388 (+ 89155us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:37.088463 (+ 75us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:37.088494 (+ 31us) tablet.cc:683] Acquiring locks for 1 operations
0806 03:51:37.088747 (+ 253us) tablet.cc:699] Row locks acquired
0806 03:51:37.088770 (+ 23us) write_op.cc:260] PREPARE: finished
0806 03:51:37.088983 (+ 213us) write_op.cc:270] Start()
0806 03:51:37.089130 (+ 147us) write_op.cc:276] Timestamp: P: 1785988297088932 usec, L: 0
0806 03:51:37.089157 (+ 27us) op_driver.cc:331] REPLICATION: starting
0806 03:51:37.089938 (+ 781us) log.cc:842] Serialized 530 byte log entry
0806 03:51:37.935114 (+845176us) op_driver.cc:444] REPLICATION: finished
0806 03:51:39.176078 (+1240964us) write_op.cc:301] APPLY: starting
0806 03:51:39.176186 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:39.176297 (+ 111us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:39.176421 (+ 124us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:39.176735 (+ 314us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:39.176905 (+ 170us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0806 03:51:39.176954 (+ 49us) write_op.cc:312] APPLY: finished
0806 03:51:39.177188 (+ 234us) log.cc:842] Serialized 23 byte log entry
0806 03:51:40.067834 (+890646us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:40.095781 (+ 27947us) write_op.cc:454] Released schema lock
0806 03:51:40.154694 (+ 58913us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:40.192262 (+ 37568us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:40.195876 (+ 3614us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":381406,"reported_tablets":2,"spinlock_wait_cycles":89387392,"child_traces":[["op",{"apply.queue_time_us":1213906,"apply.run_cpu_time_us":53354,"apply.run_wall_time_us":1046423,"num_ops":1,"prepare.queue_time_us":842697,"prepare.run_cpu_time_us":24541,"prepare.run_wall_time_us":429975,"raft.queue_time_us":264782,"raft.run_cpu_time_us":64009,"raft.run_wall_time_us":1321947,"replication_time_us":816140,"thread_start_us":2923821,"threads_started":5,"wal-append.queue_time_us":1287288,"wal-append.run_cpu_time_us":24812,"wal-append.run_wall_time_us":1479956}]]}
I20260806 03:51:40.602138 4528 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.843488 2376 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:40.844492 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1935) took 1547 ms. Trace:
I20260806 03:51:40.846115 4528 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:40.845041 2476 rpcz_store.cc:276] 0806 03:51:39.296948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:39.297108 (+ 160us) service_pool.cc:227] Handling call
0806 03:51:39.297516 (+ 408us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:39.297793 (+ 277us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:51:40.011381 (+713588us) log.cc:842] Serialized 14 byte log entry
0806 03:51:40.666239 (+654858us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:40.819695 (+153456us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:40.829490 (+ 9795us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:40.829543 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:40.844185 (+ 14642us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:40.844385 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":960533,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":73,"raft.queue_time_us":3090,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":12307,"spinlock_wait_cycles":11728640,"thread_start_us":1487204,"threads_started":2,"wal-append.queue_time_us":557047}
I20260806 03:51:40.747354 2484 rpcz_store.cc:276] 0806 03:51:38.103375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.448833 (+345458us) service_pool.cc:227] Handling call
0806 03:51:38.449221 (+ 388us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.449474 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:40.419064 (+1969590us) log.cc:842] Serialized 14 byte log entry
0806 03:51:40.561150 (+142086us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:40.577342 (+ 16192us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:40.585302 (+ 7960us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:40.593933 (+ 8631us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:40.612350 (+ 18417us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1745776,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":88,"raft.queue_time_us":145531,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":302,"thread_start_us":2068901,"threads_started":2,"wal-append.queue_time_us":195045}
I20260806 03:51:40.747818 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:40.847370 2574 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:40.752168 2483 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.878s user 0.005s sys 0.014s
I20260806 03:51:40.847618 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:40.755718 2232 catalog_manager.cc:5495] T 8db383c38ecd45f2b2cb8c46ff360638 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 1 to 2, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 2 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:51:40.849133 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 102) took 2668 ms. Trace:
I20260806 03:51:40.849448 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:40.850455 2484 raft_consensus.cc:2379] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:40.849675 2232 catalog_manager.cc:5495] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 1 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:51:40.817379 4615 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:40.760229 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 329819us
I20260806 03:51:40.757627 4915 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.761590 4871 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:40.855652 2477 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:51:40.856568 4915 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.682394 4702 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.081s
I20260806 03:51:40.764278 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:40.858090 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:40.858902 3592 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:40.764792 4743 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:40.766806 2579 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:40.863306 4664 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:51:40.769428 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:40.770840 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:40.773335 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:40.777315 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 03:51:40.737074 4627 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:40.795893 4854 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.107s user 0.056s sys 0.107s
I20260806 03:51:40.803747 4539 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.662418 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:40.791107 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.256s user 0.005s sys 0.011s
I20260806 03:51:40.801476 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:40.794391 2478 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:40.824012 4913 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:40.787577 4791 consensus_meta.cc:287] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.074s user 0.013s sys 0.012s
I20260806 03:51:40.813375 2581 raft_consensus.cc:2396] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:40.804857 4728 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.819633 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.811162 4722 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.831138 3585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36484 (request call id 265) took 1427 ms. Trace:
I20260806 03:51:40.830838 4666 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.901986 3585 rpcz_store.cc:276] 0806 03:51:39.403677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:39.405278 (+ 1601us) service_pool.cc:227] Handling call
0806 03:51:40.372239 (+966961us) spinlock_profiling.cc:243] Waited 42.4 ms on lock 0x7b5800000f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:51:40.831038 (+458799us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":97409792}
I20260806 03:51:40.830960 4926 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.902036 2572 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:40.838168 4920 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:40.589946 4792 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.268s user 0.017s sys 0.018s
I20260806 03:51:40.842686 4909 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:40.720201 4890 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.847681 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:40.849562 2483 rpcz_store.cc:276] 0806 03:51:38.180532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.463089 (+282557us) service_pool.cc:227] Handling call
0806 03:51:38.528959 (+ 65870us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:38.529240 (+ 281us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:51:39.873548 (+1344308us) log.cc:842] Serialized 14 byte log entry
0806 03:51:40.848516 (+974968us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:40.848643 (+ 127us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:51:40.848730 (+ 87us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:40.848893 (+ 163us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:40.849050 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1705214,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":70,"thread_start_us":2301015,"threads_started":2,"wal-append.queue_time_us":961942}
I20260806 03:51:40.754688 4766 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:40.759021 4910 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.762279 4367 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:40.853688 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:40.859009 2477 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:40.767071 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:40.767731 4745 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:40.847474 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:40.866638 4604 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:40.867322 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.869961 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.865231 2232 catalog_manager.cc:5495] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 2 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } }
I20260806 03:51:40.804234 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:40.876392 2577 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:40.882225 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:40.883057 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:40.884001 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:40.885648 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:51:40.885878 4539 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:40.887049 4627 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:40.887079 4702 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:40.888202 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:40.890120 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.110954 2584 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:41.073583 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:40.907143 3593 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:40.950448 2372 raft_consensus.cc:1261] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:41.115332 4662 consensus_queue.cc:1035] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260806 03:51:41.113526 4743 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260806 03:51:41.164774 2482 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:41.165858 4868 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:41.166545 4868 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:40.906421 4791 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:40.956146 4721 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.219s
I20260806 03:51:41.162030 2578 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:40.905967 4926 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.162030 2476 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
W20260806 03:51:41.071486 4943 log.cc:924] Time spent T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260806 03:51:41.081375 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:41.167989 2370 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
W20260806 03:51:41.168215 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 64296.1us
I20260806 03:51:41.169344 2377 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:40.904341 4666 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.177969 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:40.950448 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:41.175175 2537 leader_election.cc:334] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 2.151s (SENT)
I20260806 03:51:41.195182 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.178282 2474 raft_consensus.cc:2379] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:41.199671 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.074609 2331 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:41.115566 2534 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:41.189743 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 380815us
I20260806 03:51:41.116134 4831 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.086s
I20260806 03:51:41.208366 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.172632 2232 catalog_manager.cc:5495] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 2 to 4, leader changed from cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 4 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:41.176920 4791 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.179508 4920 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.171963 4622 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:41.108239 4664 consensus_queue.cc:1035] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20260806 03:51:41.177711 4604 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:41.215204 2379 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:41.326048 2375 raft_consensus.cc:2379] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:41.227375 2369 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
W20260806 03:51:41.320675 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 105080us
I20260806 03:51:41.197126 4702 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:41.328239 2478 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:41.282891 2484 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:41.283205 2368 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:41.337802 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:41.246670 2373 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:41.520426 4920 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:41.531217 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 541001us
I20260806 03:51:41.419205 2232 catalog_manager.cc:5495] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:41.532533 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 458005us
I20260806 03:51:41.536440 4920 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.280086 4627 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.539192 4920 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.540498 4627 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.540887 4539 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.542186 4831 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:41.542965 4831 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.547598 2436 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:41.548671 4939 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:41.549580 2436 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:41.549465 4939 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.550343 4792 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:41.550872 4901 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.221s user 0.001s sys 0.001s
I20260806 03:51:41.551461 4920 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:41.324656 4797 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:51:41.560712 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
W20260806 03:51:41.568779 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 453371us
I20260806 03:51:41.377311 4865 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:41.584066 2577 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
W20260806 03:51:41.585142 2480 consensus_meta.cc:287] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.772s user 0.010s sys 0.003s
I20260806 03:51:41.481105 4927 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:41.586876 2574 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.739s user 0.013s sys 0.001s
I20260806 03:51:41.601670 2574 raft_consensus.cc:2396] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:41.545573 4831 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:41.296877 4728 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.558629 2232 catalog_manager.cc:5495] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:41.609462 2478 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:51:41.583583 4766 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:41.619351 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:41.619732 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.613168 2575 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:41.601303 2480 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:41.603778 4890 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:41.645169 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:41.635404 4622 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.660219 4939 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.661209 4939 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:41.663985 4622 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.672874 2534 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:41.674192 4797 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:41.680558 4702 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.689463 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:41.614584 4920 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.692883 4920 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.625895 4865 raft_consensus.cc:686] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:41.697376 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 111) took 1011 ms. Trace:
I20260806 03:51:41.697784 2575 rpcz_store.cc:276] 0806 03:51:40.702630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:40.880910 (+178280us) service_pool.cc:227] Handling call
0806 03:51:41.608635 (+727725us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:41.632422 (+ 23787us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:41.697148 (+ 64726us) spinlock_profiling.cc:243] Waited 721 ms on lock 0x7b540028f5d0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:41.697280 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1657959808}
I20260806 03:51:41.610381 4584 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:41.700295 4584 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:41.700771 4584 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:41.702778 4604 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:41.703913 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:41.640954 4831 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260806 03:51:41.704764 2483 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:41.708061 4797 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:41.738052 2584 raft_consensus.cc:2396] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260806 03:51:41.746344 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.787426 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:41.791901 2574 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
W20260806 03:51:41.793619 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 76404.9us
I20260806 03:51:41.794346 2480 raft_consensus.cc:1261] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:41.714938 4868 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:41.802059 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.695165 4865 consensus_queue.cc:237] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:41.803750 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:41.811239 2478 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:41.609963 4792 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.813064 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.818393 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:41.820161 2484 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:41.793217 4910 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:41.823318 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.825011 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.825841 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.827322 2478 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:41.828074 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.825947 3547 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:41.850453 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:41.851493 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 47016.4us
I20260806 03:51:41.858072 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.859869 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.860391 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.860745 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.861231 3584 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:41.862649 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.863864 2578 raft_consensus.cc:2396] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:41.864151 4868 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:41.864562 4868 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:41.864930 4868 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:41.864828 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:41.866135 2373 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:41.868409 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.872866 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.876329 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:41.876477 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.648005 4927 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.877381 3587 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:41.877529 3545 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:41.880746 2534 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:41.892500 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:41.854777 4941 log.cc:924] Time spent T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260806 03:51:41.893631 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.906481 2574 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 03:51:41.906562 2576 raft_consensus.cc:2379] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:41.907306 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.908192 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.918097 2374 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:41.843988 2480 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54220
I20260806 03:51:41.923611 2480 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:41.924163 2480 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:41.896474 4702 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260806 03:51:41.925077 4664 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:41.934754 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.937912 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:41.938601 2372 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:41.940176 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.940909 2377 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:41.941311 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.944407 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.946786 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:41.947480 2476 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:41.948489 2379 raft_consensus.cc:2396] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260806 03:51:41.952440 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.953181 2379 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:51:41.953414 4792 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:41.954129 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.918471 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:41.954766 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.955130 2372 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:51:41.956117 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.956771 2369 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:41.957603 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.958707 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.918069 4906 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:41.954743 2379 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:41.961608 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:41.987684 2372 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:41.967958 3545 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:41.998971 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:42.005286 2373 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:41.933171 4927 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:42.030027 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:42.444486 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:42.444449 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:42.338208 2481 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.826s user 0.036s sys 0.102s
W20260806 03:51:42.447078 2481 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.025s user 0.039s sys 0.104s
I20260806 03:51:42.447391 2372 raft_consensus.cc:2379] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:51:42.110284 2482 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s user 0.047s sys 0.066s
W20260806 03:51:42.258687 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 372599us
W20260806 03:51:42.448621 2482 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.928s user 0.058s sys 0.067s
I20260806 03:51:42.035653 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:42.037542 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:42.450789 2373 raft_consensus.cc:2362] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4. Re-sending same reply.
I20260806 03:51:42.451239 2483 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:51:42.444694 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 895255us
I20260806 03:51:42.032092 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:42.472474 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:42.030797 2484 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
W20260806 03:51:42.125514 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 157668us
W20260806 03:51:42.405926 2478 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.010s sys 0.005s
W20260806 03:51:42.477762 2478 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.620s user 0.011s sys 0.005s
I20260806 03:51:42.048766 2480 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:41.935685 4664 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:42.497319 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:42.032296 4856 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.594s
W20260806 03:51:42.514122 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 202838us
I20260806 03:51:42.515398 4856 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:42.515654 4890 consensus_meta.cc:287] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.775s user 0.012s sys 0.000s
W20260806 03:51:42.253587 2477 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.046s user 0.008s sys 0.007s
I20260806 03:51:42.516323 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:42.516381 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:42.524392 2483 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:51:42.258554 3542 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.018s sys 0.050s
I20260806 03:51:42.517490 2436 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:42.515498 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:42.034482 4909 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.839s user 0.009s sys 0.002s
W20260806 03:51:42.260547 3585 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.022s sys 0.024s
I20260806 03:51:42.525173 2480 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:42.524550 2376 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:42.079121 4702 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:42.530849 3542 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.859s user 0.022s sys 0.055s
I20260806 03:51:42.536201 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:42.538173 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:42.539119 2369 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:42.035069 4539 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:42.542562 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:42.537302 2379 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:51:42.043133 4906 raft_consensus.cc:686] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
W20260806 03:51:42.531566 3585 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.843s user 0.025s sys 0.036s
I20260806 03:51:42.503918 4664 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:42.523823 4721 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.978s user 0.031s sys 0.073s
I20260806 03:51:42.528548 4927 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:42.524097 2477 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:42.034853 4584 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:42.086663 4868 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:42.536513 4697 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:42.565176 4721 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.021s user 0.032s sys 0.073s
I20260806 03:51:42.550446 4960 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:42.581271 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1650) took 1392 ms. Trace:
I20260806 03:51:42.517457 4856 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:42.585453 2484 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260806 03:51:42.588341 2480 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 03:51:42.589174 2575 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.729s user 0.016s sys 0.031s
I20260806 03:51:42.597143 2479 rpcz_store.cc:276] 0806 03:51:41.193406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:41.200991 (+ 7585us) service_pool.cc:227] Handling call
0806 03:51:41.503667 (+302676us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:41.536597 (+ 32930us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:41.536629 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:41.537217 (+ 588us) log.cc:842] Serialized 26 byte log entry
0806 03:51:41.897358 (+360141us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:41.897395 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:41.897479 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:42.580814 (+683335us) raft_consensus.cc:1671] finished
0806 03:51:42.580841 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:42.581024 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":359523,"threads_started":1,"wal-append.queue_time_us":861412}
W20260806 03:51:42.581982 4952 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.683s user 0.013s sys 0.004s
W20260806 03:51:42.603080 2575 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.743s user 0.016s sys 0.031s
W20260806 03:51:42.606073 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.012s sys 0.046s
I20260806 03:51:42.566881 4539 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:42.607956 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.018s sys 0.041s
W20260806 03:51:42.594300 4865 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.014s sys 0.011s
W20260806 03:51:42.607316 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.676s user 0.013s sys 0.046s
W20260806 03:51:42.542073 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 873765us
W20260806 03:51:42.608486 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.683s user 0.018s sys 0.041s
W20260806 03:51:42.609668 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 691162us
W20260806 03:51:42.610173 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 116002us
I20260806 03:51:42.597081 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 106) took 1892 ms. Trace:
I20260806 03:51:42.611774 2429 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:51:42.615370 2584 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.004s sys 0.020s
I20260806 03:51:42.616564 2479 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260806 03:51:42.612238 4745 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.003s sys 0.021s
W20260806 03:51:42.618636 2584 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.721s user 0.005s sys 0.020s
W20260806 03:51:42.609462 4865 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s user 0.014s sys 0.011s
I20260806 03:51:42.588505 4927 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:42.626333 2368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.017s sys 0.015s
W20260806 03:51:42.639372 2368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s user 0.017s sys 0.018s
W20260806 03:51:42.631498 4745 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.595s user 0.004s sys 0.021s
I20260806 03:51:42.596728 4697 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:42.649715 2479 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:42.657625 2480 raft_consensus.cc:1226] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:42.618178 2477 rpcz_store.cc:276] 0806 03:51:40.690687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:40.857717 (+167030us) service_pool.cc:227] Handling call
0806 03:51:42.582641 (+1724924us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:42.715492 2232 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.675s user 0.026s sys 0.031s
W20260806 03:51:42.718029 2232 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.677s user 0.026s sys 0.031s
I20260806 03:51:42.668339 4960 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:42.731225 2436 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:42.725457 4743 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.003s sys 0.056s
W20260806 03:51:42.748517 4963 log.cc:924] Time spent T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.151s user 0.001s sys 0.000s
I20260806 03:51:42.748740 4584 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:42.752638 4539 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:42.753770 4856 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:42.736008 4960 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:42.756628 4743 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.715s user 0.004s sys 0.057s
I20260806 03:51:42.767112 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1651) took 1566 ms. Trace:
I20260806 03:51:42.765133 4960 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:42.581686 4906 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:42.818421 4971 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:42.799113 4868 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:42.821496 4797 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:42.853411 4927 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:42.874322 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.781s user 0.000s sys 0.013s
I20260806 03:51:42.876901 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 116) took 1200 ms. Trace:
W20260806 03:51:42.870970 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 259308us
I20260806 03:51:42.823303 4743 consensus_queue.cc:1035] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
W20260806 03:51:42.860383 2329 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.018s sys 0.042s
W20260806 03:51:42.882998 2329 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.744s user 0.018s sys 0.042s
W20260806 03:51:42.877147 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.784s user 0.000s sys 0.013s
I20260806 03:51:42.877381 2478 rpcz_store.cc:276] 0806 03:51:41.676923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:41.828216 (+151293us) service_pool.cc:227] Handling call
0806 03:51:41.828543 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:41.828769 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:51:42.478289 (+649520us) log.cc:842] Serialized 14 byte log entry
0806 03:51:42.876452 (+398163us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:42.876525 (+ 73us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:42.876588 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.876668 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:42.876833 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":363370,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"thread_start_us":1018064,"threads_started":2}
I20260806 03:51:42.892982 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.896631 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.900869 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.901904 2429 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:42.880270 4927 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:42.902596 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.911890 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.916078 2429 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:42.931432 2584 raft_consensus.cc:1226] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:42.934274 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.938210 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:42.911072 4865 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:42.767676 2474 rpcz_store.cc:276] 0806 03:51:41.202364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:41.208364 (+ 6000us) service_pool.cc:227] Handling call
0806 03:51:41.685128 (+476764us) spinlock_profiling.cc:243] Waited 4.18 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:41.685351 (+ 223us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:41.690822 (+ 5471us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:41.690869 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:41.691493 (+ 624us) log.cc:842] Serialized 26 byte log entry
0806 03:51:41.882506 (+191013us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:41.895994 (+ 13488us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:41.913333 (+ 17339us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:42.766671 (+853338us) raft_consensus.cc:1671] finished
0806 03:51:42.766720 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:42.767025 (+ 305us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4911,"raft.queue_time_us":6639,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"spinlock_wait_cycles":9625472,"thread_start_us":151878,"threads_started":1,"wal-append.queue_time_us":874846}
I20260806 03:51:42.888696 4797 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:42.875342 4971 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:42.958853 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:42.964725 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 343962us
W20260806 03:51:42.966382 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 345522us
I20260806 03:51:42.968447 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 121) took 1139 ms. Trace:
I20260806 03:51:42.969290 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.970400 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:42.965160 2331 connection.cc:232] RPC call timeout handler was delayed by -1.1472922889515758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1552s for a response
I20260806 03:51:42.971096 2578 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:42.969347 2575 rpcz_store.cc:276] 0806 03:51:41.828730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:41.829268 (+ 538us) service_pool.cc:227] Handling call
0806 03:51:41.841428 (+ 12160us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:41.851492 (+ 10064us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:51:42.607330 (+755838us) log.cc:842] Serialized 14 byte log entry
0806 03:51:42.967948 (+360618us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:42.968009 (+ 61us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:42.968097 (+ 88us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.968208 (+ 111us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:42.968333 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5071,"prepare.queue_time_us":904748,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"thread_start_us":1068905,"threads_started":2}
I20260806 03:51:42.975741 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.976518 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.980166 4981 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.059s
W20260806 03:51:42.985864 4962 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.199s user 0.001s sys 0.001s
I20260806 03:51:42.959061 4927 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:42.989043 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.991603 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.972863 4865 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:42.997932 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:43.007093 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:42.999336 2538 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:43.014227 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:42.994539 4965 log.cc:924] Time spent T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.123s user 0.001s sys 0.000s
I20260806 03:51:43.000932 2584 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
W20260806 03:51:43.007642 4969 log.cc:924] Time spent T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
W20260806 03:51:43.007045 4956 log.cc:924] Time spent T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.195s user 0.000s sys 0.001s
I20260806 03:51:42.994539 5004 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:43.038269 2374 raft_consensus.cc:3032] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:43.038919 2374 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:42.996416 3585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 311) took 1815 ms. Trace:
I20260806 03:51:43.000916 4928 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:43.057803 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:43.066093 2538 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:43.064633 2374 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.066888 2372 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:51:43.066819 4856 consensus_queue.cc:1035] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:43.068079 4928 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:43.070118 2538 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:43.070545 4928 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:43.071452 4970 raft_consensus.cc:2781] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:43.071885 4970 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:43.072275 4970 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:43.072840 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:43.074083 4974 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:43.074249 4927 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:43.074519 4974 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:43.074922 4974 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:43.075145 4927 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:43.075548 4927 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:43.076565 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 49425.6us
I20260806 03:51:43.075390 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:43.072999 4971 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:43.066975 2374 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:42.996177 4993 consensus_queue.cc:1035] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.078s
I20260806 03:51:42.994019 3542 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:43.093786 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1945) took 2603 ms. Trace:
I20260806 03:51:43.075906 4927 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:43.101639 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:43.107055 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.106261 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:43.183559 4766 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.372s user 0.022s sys 0.014s
W20260806 03:51:43.205101 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 230023us
W20260806 03:51:43.303776 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 232808us
W20260806 03:51:43.288687 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 212982us
I20260806 03:51:43.056663 4996 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:43.322674 2537 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:43.305912 2373 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:43.326704 2538 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:43.268697 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 112) took 2464 ms. Trace:
I20260806 03:51:43.327531 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.267696 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 953) took 1345 ms. Trace:
I20260806 03:51:43.307538 4890 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:43.102280 4797 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:43.184139 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 194459us
W20260806 03:51:43.182089 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 456734us
I20260806 03:51:43.099763 2481 rpcz_store.cc:276] 0806 03:51:40.506785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:40.542708 (+ 35923us) service_pool.cc:227] Handling call
0806 03:51:40.601264 (+ 58556us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:41.202801 (+601537us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:42.447603 (+1244802us) log.cc:842] Serialized 14 byte log entry
0806 03:51:42.859604 (+412001us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:42.859703 (+ 99us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:42.859903 (+ 200us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:43.093429 (+233526us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b54001de2d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:43.093501 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:43.093700 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":654375,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":202136,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":12758,"spinlock_wait_cycles":1204612992,"thread_start_us":1320654,"threads_started":2,"wal-append.queue_time_us":425027}
I20260806 03:51:43.202808 4928 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:43.205315 2331 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:43.057478 3585 rpcz_store.cc:276] 0806 03:51:41.177926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:41.178717 (+ 791us) service_pool.cc:227] Handling call
0806 03:51:41.179186 (+ 469us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:41.196229 (+ 17043us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:41.201359 (+ 5130us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:41.640550 (+439191us) log.cc:842] Serialized 26 byte log entry
0806 03:51:42.572842 (+932292us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:42.588972 (+ 16130us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.596075 (+ 7103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:42.987254 (+391179us) raft_consensus.cc:1671] finished
0806 03:51:42.987288 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:42.991572 (+ 4284us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3579,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":8323,"thread_start_us":876675,"threads_started":1,"wal-append.queue_time_us":1033148}
W20260806 03:51:43.118623 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 202623us
I20260806 03:51:43.304440 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.300671 4697 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:43.339076 2538 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:43.328527 2368 rpcz_store.cc:276] 0806 03:51:41.863761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:41.868077 (+ 4316us) service_pool.cc:227] Handling call
0806 03:51:41.868488 (+ 411us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:41.868789 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:41.868814 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:41.869312 (+ 498us) log.cc:842] Serialized 26 byte log entry
0806 03:51:42.640910 (+771598us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:42.640979 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.641078 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:43.208423 (+567345us) raft_consensus.cc:1671] finished
0806 03:51:43.208461 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:43.208680 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":770500,"threads_started":1,"wal-append.queue_time_us":954081}
W20260806 03:51:43.322877 4766 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.518s user 0.024s sys 0.023s
I20260806 03:51:43.328079 2482 rpcz_store.cc:276] 0806 03:51:40.804591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:41.167135 (+362544us) service_pool.cc:227] Handling call
0806 03:51:41.167686 (+ 551us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:41.204004 (+ 36318us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:41.204054 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:41.492299 (+288245us) log.cc:842] Serialized 26 byte log entry
0806 03:51:42.449378 (+957079us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:42.449439 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.449540 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:43.268367 (+818827us) raft_consensus.cc:1671] finished
0806 03:51:43.268398 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:43.268594 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6851,"raft.run_cpu_time_us":479,"raft.run_wall_time_us":118610,"thread_start_us":928813,"threads_started":1,"wal-append.queue_time_us":1235517}
W20260806 03:51:43.347841 2332 connection.cc:232] RPC call timeout handler was delayed by -1.2885151959490031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18300000000000002s for a response
I20260806 03:51:43.327190 4978 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:43.355747 4910 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:43.358798 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.335839 4890 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:43.349468 4831 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:43.353672 4928 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:43.363350 2537 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:43.374905 2429 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:43.367045 4890 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:43.365957 4910 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:43.388504 4910 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:43.372684 5002 log.cc:924] Time spent T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.391s user 0.002s sys 0.000s
I20260806 03:51:43.390363 2429 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:43.391351 5008 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:43.391902 5008 raft_consensus.cc:686] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:43.357963 5010 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:43.392783 5010 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.392951 5008 consensus_queue.cc:237] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.395045 5010 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:43.101943 4971 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.250s
I20260806 03:51:43.395622 5010 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:43.354127 4996 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:43.396440 4996 raft_consensus.cc:686] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:43.397058 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58021.4us
I20260806 03:51:43.397502 4996 consensus_queue.cc:237] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.398307 2538 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:43.398845 2537 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:43.365559 4978 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.400573 4831 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:43.401031 4831 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:43.401399 4831 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:43.402244 4978 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:43.402833 4978 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:43.403419 4978 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:43.403755 4978 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:43.404284 5019 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:43.404716 4991 log.cc:924] Time spent T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.407s user 0.000s sys 0.001s
I20260806 03:51:43.367817 5012 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:43.412856 5012 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:43.415180 5012 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:43.361248 5013 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:43.416375 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 958) took 1478 ms. Trace:
I20260806 03:51:43.102170 4986 consensus_queue.cc:1046] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.402s
W20260806 03:51:43.416962 4998 log.cc:924] Time spent T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.340s user 0.000s sys 0.001s
W20260806 03:51:43.409219 4664 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.830s user 0.011s sys 0.007s
I20260806 03:51:43.419118 4664 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:43.407603 4959 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:43.409296 2436 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:43.416107 5013 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:43.417415 4986 raft_consensus.cc:3032] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:51:43.296867 4980 consensus_queue.cc:1046] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.405s
I20260806 03:51:43.416827 2375 rpcz_store.cc:276] 0806 03:51:41.938285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:41.940185 (+ 1900us) service_pool.cc:227] Handling call
0806 03:51:41.940602 (+ 417us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:42.030479 (+ 89877us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:42.030528 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:42.502817 (+472289us) log.cc:842] Serialized 26 byte log entry
0806 03:51:42.913822 (+411005us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:42.913877 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.913973 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:43.416098 (+502125us) raft_consensus.cc:1671] finished
0806 03:51:43.416123 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:43.416288 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":63521,"raft.queue_time_us":96118,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":137,"thread_start_us":333457,"threads_started":1,"wal-append.queue_time_us":434059}
W20260806 03:51:43.365213 5003 log.cc:924] Time spent T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.357s user 0.001s sys 0.000s
W20260806 03:51:43.365908 4997 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.301s user 0.000s sys 0.001s
I20260806 03:51:43.400332 2429 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:43.438594 4865 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:43.535629 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.491688 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.534693 2369 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260806 03:51:43.379884 4994 log.cc:924] Time spent T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.201s user 0.000s sys 0.000s
W20260806 03:51:43.538668 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 139928us
I20260806 03:51:43.536481 2373 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
W20260806 03:51:43.547621 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 183458us
I20260806 03:51:43.371644 4928 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:43.550367 5007 raft_consensus.cc:2741] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:51:43.550786 4959 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:43.553311 2436 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:43.553814 5013 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:43.555760 2368 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:43.416822 4927 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:43.549437 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 149188us
I20260806 03:51:43.579892 2577 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:43.577564 2373 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:51:43.410156 4978 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:43.424402 4960 consensus_meta.cc:287] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.600s user 0.009s sys 0.005s
I20260806 03:51:43.405922 5010 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:43.584817 2377 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:51:43.585752 2375 raft_consensus.cc:1226] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:43.589954 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.591362 2429 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:43.371484 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:43.454703 5004 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:43.589257 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.601329 2537 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:43.562503 4995 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:51:43.600857 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 498656us
I20260806 03:51:43.595011 4960 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.594942 5010 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:43.617384 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.617791 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:43.618628 2376 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:43.607934 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 54746.9us
I20260806 03:51:43.609443 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.537499 4986 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:43.612166 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:43.612336 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 212428us
I20260806 03:51:43.474710 4831 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.618124 2377 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:43.488873 4890 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:43.620246 2331 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:43.532750 4974 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.589265 5007 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:43.649046 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 56607.1us
I20260806 03:51:43.651571 4664 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:43.653061 2331 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:43.646562 4927 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:43.620950 4981 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:43.620116 4766 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:43.655915 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1667) took 1711 ms. Trace:
I20260806 03:51:43.664390 2483 rpcz_store.cc:276] 0806 03:51:41.944677 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:42.571903 (+627226us) service_pool.cc:227] Handling call
0806 03:51:42.586356 (+ 14453us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:42.588968 (+ 2612us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:42.589025 (+ 57us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:42.594798 (+ 5773us) log.cc:842] Serialized 26 byte log entry
0806 03:51:42.923036 (+328238us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:42.923064 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.923116 (+ 52us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:43.655616 (+732500us) raft_consensus.cc:1671] finished
0806 03:51:43.655641 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:43.655817 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2974,"raft.queue_time_us":2123,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":6232,"thread_start_us":308230,"threads_started":1,"wal-append.queue_time_us":414432}
I20260806 03:51:43.621506 5004 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.613997 4995 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:43.671134 4995 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:43.673828 4890 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:43.585814 4910 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.678956 4910 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:43.680601 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1665) took 1759 ms. Trace:
I20260806 03:51:43.681895 2476 rpcz_store.cc:276] 0806 03:51:41.920840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:42.551122 (+630282us) service_pool.cc:227] Handling call
0806 03:51:42.569759 (+ 18637us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:42.596800 (+ 27041us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:42.596829 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:42.606653 (+ 9824us) log.cc:842] Serialized 26 byte log entry
0806 03:51:42.939426 (+332773us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:42.939465 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.939541 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:43.673010 (+733469us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b24001374f8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d799f5cf4 00007f0d799f3dfb 00007f0d79a82559 00007f0d79a821e5 00007f0d79a81740 00007f0d7c774578 00007f0d7c774cd8 00007f0d7c74c6e8 00007f0d7c747aea 00007f0d7c746f36
0806 03:51:43.680151 (+ 7141us) raft_consensus.cc:1671] finished
0806 03:51:43.680182 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:43.680499 (+ 317us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39242112,"thread_start_us":301451,"threads_started":1,"wal-append.queue_time_us":380590}
I20260806 03:51:43.659641 2429 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:43.688993 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 120) took 1723 ms. Trace:
I20260806 03:51:43.621297 4797 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:43.689455 2484 rpcz_store.cc:276] 0806 03:51:41.965788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:42.606685 (+640897us) service_pool.cc:227] Handling call
0806 03:51:42.615509 (+ 8824us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:42.629878 (+ 14369us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:42.629930 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:42.638250 (+ 8320us) log.cc:842] Serialized 26 byte log entry
0806 03:51:42.915384 (+277134us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:42.915431 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:42.915500 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:43.688725 (+773225us) raft_consensus.cc:1671] finished
0806 03:51:43.688752 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:43.688919 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8790,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":6023,"thread_start_us":234463,"threads_started":1,"wal-append.queue_time_us":317858}
W20260806 03:51:43.703027 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 43481.7us
I20260806 03:51:43.701711 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.713295 2368 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:43.703962 2429 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:51:43.657863 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 303939us
I20260806 03:51:43.664153 4981 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:43.725284 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:43.702499 4970 consensus_meta.cc:287] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.630s user 0.005s sys 0.008s
I20260806 03:51:43.731559 2575 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:43.679957 4831 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:43.662287 4980 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:43.741628 2584 raft_consensus.cc:1226] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:43.685227 2377 raft_consensus.cc:2932] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:43.749771 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:43.689244 4766 raft_consensus.cc:686] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:43.756733 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:43.770716 2373 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:51:43.771366 2576 raft_consensus.cc:2396] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:43.718562 4910 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:43.756013 2578 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:43.756255 5004 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:43.758754 5012 raft_consensus.cc:3032] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:43.765600 4974 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:43.771873 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:43.698659 4797 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:43.685925 4697 consensus_queue.cc:1046] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260806 03:51:43.669474 4743 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260806 03:51:43.773772 3545 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:43.785372 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:43.800678 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:43.801975 2373 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:43.787847 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 70929.2us
I20260806 03:51:43.786789 2534 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:43.733312 4981 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:43.696928 4996 consensus_queue.cc:1046] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260806 03:51:43.730680 4890 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:43.737104 5008 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:43.715945 4664 consensus_queue.cc:1046] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.067s
I20260806 03:51:43.808455 2429 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:43.719833 4856 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
W20260806 03:51:43.809782 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 156830us
I20260806 03:51:43.751875 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:43.781347 4910 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:43.821290 2436 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:43.799849 4797 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:43.796265 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:43.825645 5035 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:43.827777 3542 connection.cc:232] RPC call timeout handler was delayed by -1.0092475528363138s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
I20260806 03:51:43.800225 4831 consensus_queue.cc:1046] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260806 03:51:43.743499 4970 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.830119 4970 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:43.832398 4766 consensus_queue.cc:237] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.806870 4890 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:43.806926 4906 raft_consensus.cc:3032] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:43.810607 5008 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:43.859130 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 72450.3us
I20260806 03:51:43.901304 2534 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:43.815572 4697 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:51:43.894219 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 78570.1us
I20260806 03:51:43.875203 2373 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:43.824890 4910 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:43.914057 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:43.916491 2331 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:43.753947 4980 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:43.932631 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:43.934613 5038 log.cc:924] Time spent T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260806 03:51:43.900141 5018 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:43.895450 4974 raft_consensus.cc:3032] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:43.914691 4697 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:43.900539 4906 raft_consensus.cc:729] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:43.909762 5008 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:43.938594 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:43.910302 4890 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:43.935710 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 86178.4us
I20260806 03:51:43.850449 5012 raft_consensus.cc:729] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:43.938048 2373 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:43.908502 3587 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 1 LEARNER]: Advancing to term 3
I20260806 03:51:43.947345 4974 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:43.824669 3585 raft_consensus.cc:2362] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2. Re-sending same reply.
I20260806 03:51:43.932122 4993 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:43.817727 5033 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.121s
W20260806 03:51:43.981240 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 89840.2us
W20260806 03:51:43.984133 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 210479us
I20260806 03:51:43.913242 5004 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:43.986106 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 84953.7us
I20260806 03:51:43.925182 4980 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:44.035168 5012 consensus_queue.cc:260] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:43.941344 4959 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:43.949823 4960 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:43.942123 5018 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:44.037984 2534 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:43.909492 5010 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.036846 5012 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:44.050932 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 206268us
I20260806 03:51:43.988004 4697 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:44.100330 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 47753.1us
I20260806 03:51:44.047392 4993 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:44.048691 4927 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:44.048641 5032 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:43.982438 4664 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:44.029908 5038 log.cc:924] Time spent T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260806 03:51:44.130259 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:44.102449 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:44.104475 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 176020us
I20260806 03:51:44.121996 5061 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:44.099543 5018 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:44.105865 5035 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:44.128582 2331 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:44.128706 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:44.131364 2377 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:44.136885 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:44.160475 2481 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:44.102304 5001 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:44.162200 2484 raft_consensus.cc:2362] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4. Re-sending same reply.
I20260806 03:51:44.168120 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:44.168954 2478 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:44.182477 4743 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:44.187817 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:44.123847 4993 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:44.185994 4959 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:44.102068 5004 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260806 03:51:44.193429 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:44.201270 2476 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:44.199798 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:44.207376 2482 raft_consensus.cc:1261] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:44.167217 4627 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:51:44.216483 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 88064.2us
I20260806 03:51:44.217734 2331 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:44.227324 2436 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:44.213074 5044 log.cc:924] Time spent T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20260806 03:51:44.236716 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:44.237970 2368 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:44.167119 4927 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.145069 5032 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.203153 4959 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.144394 4906 consensus_queue.cc:260] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:44.198467 4974 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.194573 5001 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:44.279884 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 70026.5us
I20260806 03:51:44.303752 2377 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
W20260806 03:51:44.298245 2376 consensus_meta.cc:287] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.672s user 0.012s sys 0.000s
I20260806 03:51:44.319780 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:44.242434 2484 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:44.221253 4627 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:44.233255 5031 log.cc:924] Time spent T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.415s user 0.002s sys 0.000s
I20260806 03:51:44.304268 2538 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:44.176525 5035 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.346195 2482 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:44.347257 2376 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
W20260806 03:51:44.350482 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 46367us
I20260806 03:51:44.351365 2538 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:44.358446 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:44.360450 2377 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:44.236426 5007 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:44.278039 5040 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.392s user 0.002s sys 0.000s
I20260806 03:51:44.159011 5061 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:44.359513 2436 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:44.382066 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:44.383129 2484 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:44.358870 4797 consensus_meta.cc:287] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.531s user 0.012s sys 0.002s
I20260806 03:51:44.261516 4983 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:44.349848 4627 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:44.284047 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:44.376657 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 159047us
I20260806 03:51:44.362323 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:44.313319 5041 log.cc:924] Time spent T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.211s user 0.000s sys 0.001s
I20260806 03:51:44.362934 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:44.360873 5056 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.215s user 0.000s sys 0.004s
I20260806 03:51:44.408232 2377 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:44.367184 4664 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:44.367323 4745 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:44.380828 4974 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:44.430467 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:44.404368 5010 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:44.439236 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:44.442710 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 83298.4us
W20260806 03:51:44.447312 4995 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.776s user 0.015s sys 0.000s
I20260806 03:51:44.412182 4797 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:44.448078 4995 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:44.450196 4797 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:44.452649 5079 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260806 03:51:44.452900 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:44.453547 2372 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:44.454231 5035 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:44.454813 5035 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:44.455332 5035 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.457228 5032 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:44.458473 5068 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 03:51:44.461331 2484 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:44.417954 4664 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:44.469766 2377 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:44.474471 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:44.475204 2484 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:44.475454 2377 raft_consensus.cc:1261] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:44.476150 2436 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:44.479835 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 441940us
I20260806 03:51:44.420902 4745 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:51:44.482542 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 300736us
I20260806 03:51:44.483930 2375 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:44.422384 5060 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:44.495815 2436 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:44.498499 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 449985us
I20260806 03:51:44.500012 2534 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:44.468262 5035 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:44.479194 4664 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:44.413206 4983 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.406102 5076 consensus_queue.cc:1035] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.109s
I20260806 03:51:44.507911 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:44.506186 4797 consensus_queue.cc:1046] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:44.501377 5004 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:44.435983 4971 consensus_queue.cc:1046] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260806 03:51:44.486224 4745 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:44.484395 5010 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:44.523900 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1647) took 1102 ms. Trace:
I20260806 03:51:44.506567 5087 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:44.524441 2370 rpcz_store.cc:276] 0806 03:51:43.421698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:43.455239 (+ 33541us) service_pool.cc:227] Handling call
0806 03:51:43.491109 (+ 35870us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:43.507679 (+ 16570us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:43.507711 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:43.556975 (+ 49264us) log.cc:842] Serialized 26 byte log entry
0806 03:51:43.690060 (+133085us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:43.690303 (+ 243us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:43.690557 (+ 254us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:44.523642 (+833085us) raft_consensus.cc:1671] finished
0806 03:51:44.523662 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:44.523823 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":132150,"threads_started":1,"wal-append.queue_time_us":378645}
I20260806 03:51:44.546952 2372 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:44.515560 4995 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:44.516619 5035 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:44.549593 2374 raft_consensus.cc:1226] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:44.558521 2484 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:44.557922 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:44.559760 2370 raft_consensus.cc:2396] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:44.523922 5010 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:44.570593 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:44.521190 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 532838us
I20260806 03:51:44.583500 5093 consensus_queue.cc:1046] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:51:44.584403 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:44.585719 5001 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:44.574107 5010 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:44.589658 4927 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:44.590394 4959 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:44.574035 2484 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
W20260806 03:51:44.574747 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 153210us
I20260806 03:51:44.703188 2373 raft_consensus.cc:1226] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:51:44.701228 2429 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:44.564872 5087 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260806 03:51:44.712855 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 151176us
I20260806 03:51:44.553421 5035 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:51:44.728848 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 60386.5us
W20260806 03:51:44.733896 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 224980us
I20260806 03:51:44.738381 3547 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:44.557919 4743 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.165s
I20260806 03:51:44.749563 2534 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:44.762241 2436 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:44.727773 5035 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:44.765542 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 257692us
I20260806 03:51:44.767529 2429 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:44.727388 5093 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:44.768779 2331 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:44.774251 2429 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:44.742570 4959 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:44.780951 2436 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:44.781533 2429 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:44.771559 5032 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:44.793183 2538 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:44.807564 2534 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:44.752673 4996 consensus_queue.cc:1046] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260806 03:51:44.811599 2538 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:44.784233 4995 raft_consensus.cc:2781] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:44.814703 4865 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:44.816418 5076 raft_consensus.cc:2781] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:44.789878 5007 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:44.783211 4959 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:44.783989 5004 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:51:44.824009 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 86085.6us
W20260806 03:51:44.825469 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 44636.9us
W20260806 03:51:44.850556 2476 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.643s user 0.020s sys 0.002s
I20260806 03:51:44.851460 2476 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:51:44.853176 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:44.806105 5032 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:44.860091 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 78598.7us
I20260806 03:51:44.819181 4971 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
W20260806 03:51:44.848227 4910 consensus_meta.cc:287] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.912s user 0.021s sys 0.000s
I20260806 03:51:44.862439 2429 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:44.820444 4995 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:44.879709 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:44.855057 4959 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:44.878084 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:44.867249 4865 raft_consensus.cc:686] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:44.861191 5007 raft_consensus.cc:686] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:51:44.895125 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 56375us
I20260806 03:51:44.788800 4986 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.870384 5032 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:44.912487 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 49886.1us
I20260806 03:51:45.088197 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.256191 2479 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:45.256805 2429 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:51:45.258024 4890 consensus_meta.cc:287] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.205s user 0.012s sys 0.004s
I20260806 03:51:45.108757 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.047922 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.047857 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.108368 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.048461 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:45.264972 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 184485us
I20260806 03:51:45.017652 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:45.268525 2572 raft_consensus.cc:2379] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:45.019974 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.159997 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:44.910521 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:45.196182 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 426278us
W20260806 03:51:45.036950 2368 consensus_meta.cc:287] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.798s user 0.016s sys 0.007s
W20260806 03:51:45.046733 2482 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.700s user 0.018s sys 0.002s
I20260806 03:51:45.048705 2584 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:44.989984 2480 raft_consensus.cc:1226] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 03:51:44.975028 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:45.123492 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1669) took 1506 ms. Trace:
I20260806 03:51:44.947192 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:45.099608 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 269290us
I20260806 03:51:45.015900 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:44.873721 4983 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:45.187846 4996 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:45.090525 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:45.211649 4971 raft_consensus.cc:686] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:44.954763 5013 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:51:45.269564 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 180042us
W20260806 03:51:45.192824 5104 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.316s user 0.001s sys 0.000s
W20260806 03:51:45.074488 4978 consensus_meta.cc:287] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.474s user 0.019s sys 0.000s
W20260806 03:51:45.229547 5100 log.cc:924] Time spent T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.251s user 0.000s sys 0.002s
W20260806 03:51:44.990401 4981 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.175s user 0.015s sys 0.023s
W20260806 03:51:45.122666 4974 consensus_meta.cc:287] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.685s user 0.010s sys 0.005s
W20260806 03:51:45.229025 5010 consensus_meta.cc:287] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.640s user 0.018s sys 0.002s
W20260806 03:51:45.123193 5088 log.cc:924] Time spent T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.353s user 0.002s sys 0.000s
W20260806 03:51:45.161871 5105 log.cc:924] Time spent T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.136s user 0.000s sys 0.001s
I20260806 03:51:45.273355 2331 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:44.977916 3587 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.877s user 0.017s sys 0.004s
I20260806 03:51:44.903272 4995 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:44.976699 4928 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.392s user 0.020s sys 0.006s
I20260806 03:51:45.278296 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:45.278453 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:44.991026 5109 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:44.964370 5118 raft_consensus.cc:3032] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:45.108585 4970 raft_consensus.cc:3032] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:44.978188 5116 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:44.964103 5067 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:45.134848 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:44.975641 5061 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:44.913239 4831 consensus_queue.cc:1035] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.468s
I20260806 03:51:44.906265 4910 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:44.992244 5111 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.473s
I20260806 03:51:45.280375 2538 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:45.565044 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:45.594540 2478 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:45.105391 5019 raft_consensus.cc:3032] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:45.566917 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.433871 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.567183 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 985) took 1715 ms. Trace:
I20260806 03:51:45.382069 2576 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:45.598061 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1687) took 1459 ms. Trace:
I20260806 03:51:45.596634 2372 rpcz_store.cc:276] 0806 03:51:43.852281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.558472 (+706191us) service_pool.cc:227] Handling call
0806 03:51:44.559734 (+ 1262us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:44.560356 (+ 622us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:44.560385 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:44.560957 (+ 572us) log.cc:842] Serialized 26 byte log entry
0806 03:51:45.223234 (+662277us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:45.223300 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:45.223421 (+ 121us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:45.523646 (+300225us) raft_consensus.cc:1671] finished
0806 03:51:45.523671 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:45.567077 (+ 43406us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":484920,"threads_started":1,"wal-append.queue_time_us":946640}
I20260806 03:51:45.599238 2482 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:44.978456 5119 raft_consensus.cc:2781] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:45.601955 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.603561 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 125) took 1397 ms. Trace:
I20260806 03:51:45.610708 2482 rpcz_store.cc:276] 0806 03:51:44.206126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.214130 (+ 8004us) service_pool.cc:227] Handling call
0806 03:51:45.603485 (+1389355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:45.611738 2374 rpcz_store.cc:276] 0806 03:51:43.617299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.456506 (+839207us) service_pool.cc:227] Handling call
0806 03:51:44.549388 (+ 92882us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:44.553536 (+ 4148us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:45.123187 (+569651us) spinlock_profiling.cc:243] Waited 92.5 ms on lock 0x7b5400012cd0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:45.123404 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":212661632}
I20260806 03:51:45.612900 2482 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:51:45.598542 2480 rpcz_store.cc:276] 0806 03:51:44.138626 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.147088 (+ 8462us) service_pool.cc:227] Handling call
0806 03:51:44.989427 (+842339us) spinlock_profiling.cc:243] Waited 1.34 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:44.989696 (+ 269us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:45.597764 (+608068us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:45.598000 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3079552}
I20260806 03:51:45.380538 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:45.258694 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 124) took 1099 ms. Trace:
I20260806 03:51:45.596092 2368 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:51:45.620082 2479 rpcz_store.cc:276] 0806 03:51:44.159596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.257455 (+1097859us) service_pool.cc:227] Handling call
0806 03:51:45.258000 (+ 545us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:45.258254 (+ 254us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:45.258291 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:45.258354 (+ 63us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:45.258412 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:45.258479 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:45.258616 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:45.428646 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:45.623281 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 349994us
I20260806 03:51:45.624387 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1695) took 1422 ms. Trace:
I20260806 03:51:45.624866 2482 rpcz_store.cc:276] 0806 03:51:44.202513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.612066 (+1409553us) service_pool.cc:227] Handling call
0806 03:51:45.612593 (+ 527us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:45.624063 (+ 11470us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:45.624316 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:45.628813 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 348553us
I20260806 03:51:45.631628 3591 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
W20260806 03:51:45.573208 5123 log.cc:924] Time spent T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.313s user 0.000s sys 0.000s
I20260806 03:51:45.633162 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.633661 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:45.447113 4993 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.240s user 0.014s sys 0.000s
W20260806 03:51:45.444854 5004 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.013s sys 0.018s
I20260806 03:51:45.747112 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:45.645932 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:45.520785 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:45.746837 4993 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:45.749403 5127 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:45.749611 2370 raft_consensus.cc:1226] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:51:45.749891 5127 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:45.750277 5127 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:45.751787 4993 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:45.752966 5136 log.cc:924] Time spent T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20260806 03:51:45.753165 4993 raft_consensus.cc:2741] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260806 03:51:45.755055 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 109810us
I20260806 03:51:45.616155 5109 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:45.603695 5019 raft_consensus.cc:729] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:45.743026 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:45.380903 5076 raft_consensus.cc:686] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:45.280078 5102 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:45.743845 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:45.641978 2579 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:51:45.597198 5067 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:45.785780 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:45.786245 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:45.787278 2483 raft_consensus.cc:2396] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:51:45.787710 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1680) took 1659 ms. Trace:
I20260806 03:51:45.788048 2483 rpcz_store.cc:276] 0806 03:51:44.134421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.147261 (+ 12840us) service_pool.cc:227] Handling call
0806 03:51:45.786057 (+1638796us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:51:45.787638 (+ 1581us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2877696}
I20260806 03:51:45.792505 2538 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:45.629839 3586 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:45.745080 2373 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:51:45.275259 5114 consensus_queue.cc:1046] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.480s
I20260806 03:51:45.746127 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 127) took 1483 ms. Trace:
I20260806 03:51:45.812140 2480 rpcz_store.cc:276] 0806 03:51:44.262834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.614053 (+1351219us) service_pool.cc:227] Handling call
0806 03:51:45.745488 (+131435us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:45.745704 (+ 216us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:45.745740 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:45.745801 (+ 61us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:51:45.745871 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:45.745945 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:45.746062 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:45.632656 3594 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
W20260806 03:51:45.636250 2368 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 992) took 1712 ms (client timeout 1894 ms). Trace:
W20260806 03:51:45.813351 2368 rpcz_store.cc:269] 0806 03:51:43.909072 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.219945 (+310873us) service_pool.cc:227] Handling call
0806 03:51:45.621313 (+1401368us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:45.747825 5004 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.867s user 0.019s sys 0.020s
I20260806 03:51:45.752686 5112 consensus_queue.cc:1046] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:51:45.746645 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 489885us
I20260806 03:51:45.639638 5116 raft_consensus.cc:686] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:45.593201 5118 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:45.754583 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:45.258688 4890 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:45.477666 4865 consensus_queue.cc:237] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:45.593298 4970 raft_consensus.cc:729] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:45.761476 2484 raft_consensus.cc:1226] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:45.616676 4960 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:45.747381 5124 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:45.743541 5018 consensus_meta.cc:287] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.576s user 0.009s sys 0.017s
W20260806 03:51:45.646206 4664 consensus_meta.cc:287] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.122s user 0.011s sys 0.008s
I20260806 03:51:45.635319 5013 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:45.566023 4539 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:45.610181 5119 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:45.766009 3587 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:51:45.600888 5010 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:45.617448 5007 consensus_queue.cc:237] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:45.816550 2429 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:45.823284 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:45.620463 4978 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:45.628171 4996 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:45.843047 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:45.618351 4981 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:45.858197 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1700) took 1556 ms. Trace:
I20260806 03:51:45.791934 5067 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:45.817030 5004 raft_consensus.cc:686] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:45.615039 4928 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:45.894176 5126 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:45.911247 4974 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:51:46.057392 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:46.057570 2476 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:45.879348 4971 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:46.056713 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 240311us
I20260806 03:51:46.058605 5010 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:45.991514 5018 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:46.057771 5121 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:46.060178 5119 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:46.061493 4996 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:46.061690 2575 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:46.062274 5013 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.062644 4978 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:46.064827 3587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 321) took 2341 ms. Trace:
I20260806 03:51:46.065272 3587 rpcz_store.cc:276] 0806 03:51:43.723393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:43.740067 (+ 16674us) service_pool.cc:227] Handling call
0806 03:51:43.824126 (+ 84059us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:46.064466 (+2240340us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.064720 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:46.066069 5018 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:46.059062 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:46.020828 5076 consensus_queue.cc:237] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.067370 2483 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:45.923904 5124 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:46.063160 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:46.056180 4664 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.068871 2480 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:46.069471 2583 raft_consensus.cc:2379] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:46.069540 4981 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.069842 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:46.012964 2474 raft_consensus.cc:1226] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:51:46.070865 2480 raft_consensus.cc:2396] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:51:46.071424 3590 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 03:51:45.923564 5019 consensus_queue.cc:260] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.072482 4928 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.072634 5019 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:45.918694 5118 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:46.034941 4970 consensus_queue.cc:260] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:46.016418 2538 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.785s (SENT) [suppressed 43 similar messages]
I20260806 03:51:46.074241 4970 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:46.056586 5126 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.068142 3584 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 03:51:46.075306 4664 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:46.070811 3591 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:46.020440 4697 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.891s user 0.017s sys 0.001s
I20260806 03:51:46.076506 5124 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.080103 4697 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:46.070082 5013 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.082325 4697 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.056314 2484 rpcz_store.cc:276] 0806 03:51:44.301967 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.638674 (+1336707us) service_pool.cc:227] Handling call
0806 03:51:45.761265 (+122591us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:45.857704 (+ 96439us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:45.858091 (+ 387us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:46.083895 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:46.084692 2484 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:46.087827 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40712 (request call id 490) took 2504 ms. Trace:
I20260806 03:51:46.088361 2368 rpcz_store.cc:276] 0806 03:51:43.558119 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.858405 (+2300286us) service_pool.cc:227] Handling call
0806 03:51:46.032645 (+174240us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:46.090585 5051 log.cc:924] Time spent T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 1.890s user 0.007s sys 0.000s
I20260806 03:51:46.092826 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1659) took 2540 ms. Trace:
I20260806 03:51:46.093230 2369 rpcz_store.cc:276] 0806 03:51:43.552758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:43.616503 (+ 63745us) service_pool.cc:227] Handling call
0806 03:51:43.656617 (+ 40114us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:43.682044 (+ 25427us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:43.682083 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:43.682648 (+ 565us) log.cc:842] Serialized 26 byte log entry
0806 03:51:43.893957 (+211309us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:43.893996 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:43.894078 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:46.090187 (+2196109us) spinlock_profiling.cc:243] Waited 4.16 ms on lock 0x7b24001bd9b8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d799f5cf4 00007f0d799f3dfb 00007f0d79a82559 00007f0d79a821e5 00007f0d79a8182d 00007f0d7c73046e 00007f0d7c747c0c 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551
0806 03:51:46.092447 (+ 2260us) raft_consensus.cc:1671] finished
0806 03:51:46.092476 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:46.092747 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4684,"raft.queue_time_us":20128,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":29330,"spinlock_wait_cycles":9590656,"thread_start_us":210771,"threads_started":1,"wal-append.queue_time_us":424264}
W20260806 03:51:46.096887 5059 log.cc:924] Time spent T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 1.939s user 0.002s sys 0.005s
I20260806 03:51:46.097151 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:45.845144 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1685) took 1707 ms. Trace:
I20260806 03:51:45.831588 5141 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:46.099294 5053 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:51:46.099567 5141 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:46.102610 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1672) took 2401 ms. Trace:
I20260806 03:51:46.103101 2377 rpcz_store.cc:276] 0806 03:51:43.701300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.487910 (+786610us) service_pool.cc:227] Handling call
0806 03:51:44.488599 (+ 689us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:44.708499 (+219900us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:44.708549 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:44.709227 (+ 678us) log.cc:842] Serialized 26 byte log entry
0806 03:51:45.092642 (+383415us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:45.092705 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:45.092814 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:46.102323 (+1009509us) raft_consensus.cc:1671] finished
0806 03:51:46.102350 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:46.102498 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3804,"raft.queue_time_us":220137,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":7700,"thread_start_us":419022,"threads_started":2,"wal-append.queue_time_us":392354}
I20260806 03:51:46.110903 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.077051 5126 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:46.108242 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1665) took 2553 ms. Trace:
I20260806 03:51:46.100121 5053 raft_consensus.cc:686] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:46.098836 2477 rpcz_store.cc:276] 0806 03:51:44.137715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.146298 (+ 8583us) service_pool.cc:227] Handling call
0806 03:51:44.206260 (+ 59962us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:44.233647 (+ 27387us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:44.236167 (+ 2520us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:44.341681 (+105514us) log.cc:842] Serialized 26 byte log entry
0806 03:51:44.519392 (+177711us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:44.531169 (+ 11777us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:44.542824 (+ 11655us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:45.813232 (+1270408us) raft_consensus.cc:1671] finished
0806 03:51:45.816697 (+ 3465us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:45.834536 (+ 17839us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":159232,"threads_started":1,"wal-append.queue_time_us":400496}
W20260806 03:51:46.120402 3542 leader_election.cc:334] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.144s (SENT)
I20260806 03:51:46.120299 5053 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:46.120894 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:46.121100 3542 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:45.835878 5102 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.116375 2376 rpcz_store.cc:276] 0806 03:51:43.554715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.359228 (+804513us) service_pool.cc:227] Handling call
0806 03:51:44.359597 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:44.359873 (+ 276us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:44.359899 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:44.366852 (+ 6953us) log.cc:842] Serialized 26 byte log entry
0806 03:51:44.575520 (+208668us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:44.575575 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:44.575677 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:46.107781 (+1532104us) raft_consensus.cc:1671] finished
0806 03:51:46.107805 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:46.108168 (+ 363us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":208142,"threads_started":1,"wal-append.queue_time_us":649250}
W20260806 03:51:46.108500 5012 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.989s user 0.013s sys 0.014s
I20260806 03:51:46.122326 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1664) took 2568 ms. Trace:
I20260806 03:51:46.118314 2377 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:46.202839 3591 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:46.192535 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:46.192706 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 72583us
W20260806 03:51:46.209450 3542 leader_election.cc:334] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.188s (SENT)
I20260806 03:51:46.210413 3542 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:46.121573 2477 raft_consensus.cc:2362] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4. Re-sending same reply.
I20260806 03:51:46.193552 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1699) took 1891 ms. Trace:
I20260806 03:51:46.104702 5121 raft_consensus.cc:686] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:46.211468 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.212508 2376 raft_consensus.cc:2396] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:46.211512 2482 rpcz_store.cc:276] 0806 03:51:44.301718 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.626756 (+1325038us) service_pool.cc:227] Handling call
0806 03:51:45.743787 (+117031us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:45.744088 (+ 301us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:45.849436 (+105348us) log.cc:842] Serialized 14 byte log entry
0806 03:51:46.193139 (+343703us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:46.193204 (+ 65us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:46.193259 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.193335 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:46.193482 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":25056,"thread_start_us":424775,"threads_started":2}
I20260806 03:51:46.200487 2379 rpcz_store.cc:276] 0806 03:51:43.554365 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:43.820993 (+266628us) service_pool.cc:227] Handling call
0806 03:51:43.825963 (+ 4970us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:43.826197 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:43.826220 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:43.826727 (+ 507us) log.cc:842] Serialized 26 byte log entry
0806 03:51:43.920543 (+ 93816us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:43.920593 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:43.920676 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:46.096107 (+2175431us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:46.100606 (+ 4499us) raft_consensus.cc:1671] finished
0806 03:51:46.100631 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:46.122242 (+ 21611us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":33361792,"thread_start_us":93399,"threads_started":1,"wal-append.queue_time_us":212710}
I20260806 03:51:46.100112 5141 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:46.123456 5008 consensus_meta.cc:287] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 2.070s user 0.016s sys 0.008s
I20260806 03:51:46.222083 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:46.112706 4539 raft_consensus.cc:686] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:46.123080 4910 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:46.201071 4745 consensus_meta.cc:287] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.652s user 0.021s sys 0.000s
W20260806 03:51:46.228605 2478 consensus_meta.cc:287] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.633s user 0.012s sys 0.000s
I20260806 03:51:46.230695 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:46.232206 4627 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.813s user 0.014s sys 0.018s
I20260806 03:51:46.250111 4627 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.250321 5121 consensus_queue.cc:237] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:46.252714 4539 consensus_queue.cc:237] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.207878 5126 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:46.256515 5109 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.145319 5018 consensus_queue.cc:1046] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.097s
I20260806 03:51:46.256944 4890 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.219339 4993 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:46.256755 5126 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.256186 5061 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:46.257678 4993 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.258188 3590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 320) took 2540 ms. Trace:
I20260806 03:51:46.259104 5126 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:46.258630 3590 rpcz_store.cc:276] 0806 03:51:43.718094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:43.718367 (+ 273us) service_pool.cc:227] Handling call
0806 03:51:43.743629 (+ 25262us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:46.071863 (+2328234us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.257831 (+185968us) spinlock_profiling.cc:243] Waited 2.33 s on lock 0x7b54004a2254. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:46.258110 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5352881024}
I20260806 03:51:46.261618 2374 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:46.192310 4927 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:46.197781 5124 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:46.262135 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 52730.9us
I20260806 03:51:46.273545 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:46.240379 2478 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:46.276255 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 259883us
W20260806 03:51:46.276307 2478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1694) took 2108 ms (client timeout 1785 ms). Trace:
I20260806 03:51:46.277221 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 962) took 1126 ms. Trace:
I20260806 03:51:46.281118 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:46.279852 2478 rpcz_store.cc:269] 0806 03:51:44.168555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.542590 (+1374035us) service_pool.cc:227] Handling call
0806 03:51:46.276138 (+733548us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:46.284195 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:46.282393 2368 raft_consensus.cc:2379] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:46.224848 5154 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:46.260824 5004 consensus_queue.cc:237] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.321447 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:46.326815 2478 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:46.331219 2376 raft_consensus.cc:1226] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:46.240327 5008 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.262759 4745 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:46.086251 5116 consensus_queue.cc:237] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:46.332707 5140 log.cc:924] Time spent T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.242s user 0.001s sys 0.001s
I20260806 03:51:46.274360 4927 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.360845 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:46.336256 4980 consensus_meta.cc:287] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 2.220s user 0.015s sys 0.014s
I20260806 03:51:46.372557 2481 raft_consensus.cc:1226] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 03:51:46.303956 4664 consensus_queue.cc:1046] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.060s
I20260806 03:51:46.374461 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.375250 2376 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:46.376121 2581 raft_consensus.cc:2379] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:46.344499 4627 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:46.384445 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 97479.7us
W20260806 03:51:46.370057 5032 consensus_meta.cc:287] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.093s user 0.016s sys 0.002s
I20260806 03:51:46.408054 2482 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:46.450435 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.454857 2368 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:46.302613 5154 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.381912 4980 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.479820 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.481027 2368 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:46.411784 5032 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:46.473160 5102 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:46.493057 2372 raft_consensus.cc:1226] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:46.506669 4927 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:46.509752 2332 leader_election.cc:334] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.736s (SENT)
I20260806 03:51:46.510259 2332 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:46.513386 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.522866 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:46.524300 2483 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:46.525513 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:46.527323 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:46.528205 4993 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.528460 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:46.529659 2332 leader_election.cc:334] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.547s (SENT)
I20260806 03:51:46.530314 4627 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:46.536860 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:46.557749 2584 raft_consensus.cc:2379] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:46.537143 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:46.542711 5171 log.cc:924] Time spent T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.110s user 0.000s sys 0.001s
I20260806 03:51:46.554009 2332 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:46.555186 2480 raft_consensus.cc:2396] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260806 03:51:46.565304 2332 leader_election.cc:334] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.562s (SENT)
I20260806 03:51:46.566309 2332 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:46.567525 5109 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:46.578186 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.579324 2369 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
W20260806 03:51:46.581966 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 247657us
I20260806 03:51:46.599278 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:46.601001 5164 log.cc:924] Time spent T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.163s user 0.001s sys 0.000s
W20260806 03:51:46.601720 5162 log.cc:924] Time spent T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.189s user 0.000s sys 0.001s
I20260806 03:51:46.602514 4980 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.582017 5008 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:46.620368 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:46.623847 5032 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:46.558830 4745 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:46.632637 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:46.731643 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:46.573779 4993 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:46.640595 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1668) took 3049 ms. Trace:
I20260806 03:51:46.639724 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:46.646529 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.648205 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:46.656901 2581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 960) took 1597 ms (client timeout 1798 ms). Trace:
I20260806 03:51:46.662374 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1692) took 2521 ms. Trace:
W20260806 03:51:46.664921 5050 log.cc:924] Time spent T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 2.545s user 0.000s sys 0.004s
I20260806 03:51:46.669075 3584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 319) took 2964 ms. Trace:
I20260806 03:51:46.670868 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 957) took 1616 ms. Trace:
I20260806 03:51:46.672770 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 995) took 2320 ms. Trace:
W20260806 03:51:46.674355 5070 log.cc:924] Time spent T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 2.309s user 0.001s sys 0.002s
I20260806 03:51:46.678618 5102 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:46.732928 2584 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
W20260806 03:51:47.126273 2332 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.013s sys 0.021s
W20260806 03:51:47.253636 5010 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.745s user 0.003s sys 0.025s
W20260806 03:51:47.419240 2332 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.848s user 0.019s sys 0.026s
W20260806 03:51:47.419391 5010 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s user 0.005s sys 0.028s
W20260806 03:51:47.419638 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 848418us
W20260806 03:51:46.795518 3542 leader_election.cc:334] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.095s (SENT)
I20260806 03:51:47.421808 3542 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:47.422991 2376 raft_consensus.cc:2379] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:47.424346 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:47.424468 2574 rpcz_store.cc:276] 0806 03:51:45.054751 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.054904 (+ 153us) service_pool.cc:227] Handling call
0806 03:51:46.670613 (+1615709us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540004ae54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:46.670789 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2510339072}
I20260806 03:51:47.425132 2376 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
W20260806 03:51:47.425503 2332 leader_election.cc:334] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.597s (SENT)
I20260806 03:51:47.425824 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:47.426046 2332 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:47.426373 2574 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260806 03:51:47.426553 3590 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:47.426685 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:47.428125 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:47.428792 2369 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:47.430491 2481 rpcz_store.cc:276] 0806 03:51:44.140653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.231211 (+ 90558us) service_pool.cc:227] Handling call
0806 03:51:44.231850 (+ 639us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:46.373193 (+2141343us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.662065 (+288872us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b54001216d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:46.662291 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4922411904}
I20260806 03:51:47.430989 3587 raft_consensus.cc:1226] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:47.431269 2375 rpcz_store.cc:276] 0806 03:51:43.590801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.438898 (+848097us) service_pool.cc:227] Handling call
0806 03:51:44.481388 (+ 42490us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:44.489735 (+ 8347us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.640309 (+2150574us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b54004281d0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:46.640497 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7605120}
W20260806 03:51:47.364158 4959 consensus_meta.cc:287] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 2.415s user 0.013s sys 0.016s
I20260806 03:51:46.843495 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:47.503499 2480 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:46.732776 5128 consensus_queue.cc:1046] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.701s
I20260806 03:51:47.021857 5154 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:47.027570 5035 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 2.245s user 0.020s sys 0.020s
I20260806 03:51:46.961824 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1698) took 2660 ms. Trace:
I20260806 03:51:46.740099 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 991) took 2831 ms. Trace:
W20260806 03:51:47.498093 4745 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s user 0.011s sys 0.016s
I20260806 03:51:47.418578 3584 rpcz_store.cc:276] 0806 03:51:43.704890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:43.724212 (+ 19322us) service_pool.cc:227] Handling call
0806 03:51:43.757596 (+ 33384us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:46.075093 (+2317497us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.668633 (+593540us) spinlock_profiling.cc:243] Waited 2.31 s on lock 0x7b540070d0d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:46.668968 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5312997632}
I20260806 03:51:47.432346 2372 rpcz_store.cc:276] 0806 03:51:44.352703 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.615457 (+1262754us) service_pool.cc:227] Handling call
0806 03:51:45.745263 (+129806us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:46.498802 (+753539us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.672499 (+173697us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b54003e9ed4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:46.672698 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1715851776}
W20260806 03:51:47.086086 4995 consensus_meta.cc:287] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.446s user 0.019s sys 0.008s
I20260806 03:51:46.907884 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:47.520359 2483 raft_consensus.cc:2396] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:47.423000 4906 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:47.020231 5116 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.057s sys 0.111s
I20260806 03:51:47.426820 2574 pending_rounds.cc:76] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 1
W20260806 03:51:47.486733 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 61259us
I20260806 03:51:46.785897 5018 raft_consensus.cc:3032] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
W20260806 03:51:47.311368 2576 consensus_meta.cc:287] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.956s user 0.013s sys 0.012s
I20260806 03:51:47.495453 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:46.785693 5156 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:47.498297 2478 consensus_meta.cc:287] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.171s user 0.017s sys 0.022s
I20260806 03:51:47.603470 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:47.432579 2581 rpcz_store.cc:269] 0806 03:51:45.059055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.277398 (+218343us) service_pool.cc:227] Handling call
0806 03:51:46.656562 (+1379164us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b540013bad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:46.656762 (+ 200us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1792813312}
I20260806 03:51:47.505152 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:47.604426 2372 raft_consensus.cc:2379] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:47.507026 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:47.487845 5093 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.809s user 0.009s sys 0.022s
W20260806 03:51:47.497272 5001 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.050s user 0.006s sys 0.023s
I20260806 03:51:46.738758 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 986) took 2886 ms. Trace:
I20260806 03:51:47.610292 2370 rpcz_store.cc:276] 0806 03:51:43.852691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.585577 (+732886us) service_pool.cc:227] Handling call
0806 03:51:44.585982 (+ 405us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:45.750067 (+1164085us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.738375 (+988308us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b540014e954. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:46.738667 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2675520256}
I20260806 03:51:47.513806 2373 rpcz_store.cc:276] 0806 03:51:43.908761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.719000 (+810239us) service_pool.cc:227] Handling call
0806 03:51:44.719565 (+ 565us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:45.812292 (+1092727us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.739726 (+927434us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b54001eafd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:46.740003 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2171027840}
I20260806 03:51:47.516010 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:47.612095 2375 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:47.501322 5102 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:47.496701 4890 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:47.521209 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:47.522655 2332 leader_election.cc:334] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.749s (SENT)
I20260806 03:51:47.510111 5111 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 03:51:47.516811 4745 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s user 0.011s sys 0.017s
I20260806 03:51:47.625551 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:47.610060 5001 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.175s user 0.006s sys 0.024s
W20260806 03:51:47.523510 5116 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.015s user 0.060s sys 0.116s
I20260806 03:51:47.521883 2574 pending_rounds.cc:97] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 03:51:47.638978 2574 raft_consensus.cc:2866] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:47.639333 5130 raft_consensus.cc:3032] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:47.640002 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 976) took 1078 ms. Trace:
I20260806 03:51:47.640647 2574 rpcz_store.cc:276] 0806 03:51:46.561895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.425672 (+863777us) service_pool.cc:227] Handling call
0806 03:51:47.426031 (+ 359us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:47.639655 (+213624us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.639925 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:47.643440 2577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 950) took 2758 ms (client timeout 1936 ms). Trace:
I20260806 03:51:47.501441 4993 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 03:51:47.523036 5127 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.772s user 0.013s sys 0.013s
I20260806 03:51:47.645481 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1683) took 3508 ms. Trace:
I20260806 03:51:47.645838 5127 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:47.646124 2474 rpcz_store.cc:276] 0806 03:51:44.136777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.157891 (+ 21114us) service_pool.cc:227] Handling call
0806 03:51:44.236145 (+ 78254us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:46.070753 (+1834608us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.645190 (+1574437us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b54001dd8d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:47.645409 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3862545792}
I20260806 03:51:47.647241 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 996) took 3294 ms. Trace:
I20260806 03:51:47.647671 2374 rpcz_store.cc:276] 0806 03:51:44.353224 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.622138 (+1268914us) service_pool.cc:227] Handling call
0806 03:51:45.745252 (+123114us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:46.263937 (+518685us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.646986 (+1383049us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b5400012a54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:47.647169 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1187016576}
W20260806 03:51:47.648617 2583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 963) took 2376 ms (client timeout 2861 ms). Trace:
I20260806 03:51:47.602990 2376 raft_consensus.cc:2379] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:47.649286 5127 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:47.649788 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1679) took 1003 ms. Trace:
W20260806 03:51:47.649034 2583 rpcz_store.cc:269] 0806 03:51:45.272427 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.275040 (+ 2613us) service_pool.cc:227] Handling call
0806 03:51:47.648410 (+2373370us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54004461d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:47.648550 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1088289024}
I20260806 03:51:47.650159 2376 rpcz_store.cc:276] 0806 03:51:46.646351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.425814 (+779463us) service_pool.cc:227] Handling call
0806 03:51:47.649720 (+223906us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.651257 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:47.518622 5154 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:47.639828 5130 raft_consensus.cc:729] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:47.652581 5154 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:47.280454 4856 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 3.138s user 0.018s sys 0.014s
I20260806 03:51:47.649122 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:47.654525 4856 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 3.531s user 0.020s sys 0.016s
I20260806 03:51:47.655064 2374 raft_consensus.cc:2379] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:47.601792 2576 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:51:47.603574 2478 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:47.656579 2478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1717) took 1377 ms (client timeout 1648 ms). Trace:
W20260806 03:51:47.656948 2478 rpcz_store.cc:269] 0806 03:51:46.278887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:46.321065 (+ 42178us) service_pool.cc:227] Handling call
0806 03:51:47.656507 (+1335442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.658030 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:47.643966 2577 rpcz_store.cc:269] 0806 03:51:44.885449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.885680 (+ 231us) service_pool.cc:227] Handling call
0806 03:51:47.643138 (+2757458us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b540028fad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:47.643316 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4695996672}
I20260806 03:51:47.658676 2478 raft_consensus.cc:2362] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5. Re-sending same reply.
I20260806 03:51:47.518805 5035 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:47.647564 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:47.602664 4995 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:47.665346 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:47.666050 2374 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:47.616421 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:47.667200 2373 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:47.668423 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:47.617163 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:47.669711 2377 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:47.671164 2377 pending_rounds.cc:76] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 2
I20260806 03:51:47.528630 5018 raft_consensus.cc:729] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:47.623232 2332 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:47.609809 5093 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.931s user 0.009s sys 0.023s
I20260806 03:51:47.513731 2479 rpcz_store.cc:276] 0806 03:51:44.301311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.621985 (+1320674us) service_pool.cc:227] Handling call
0806 03:51:45.745991 (+124006us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:45.772210 (+ 26219us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:45.772262 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:45.787427 (+ 15165us) log.cc:842] Serialized 26 byte log entry
0806 03:51:46.284922 (+497495us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:46.284985 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.285078 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:46.961487 (+676409us) raft_consensus.cc:1671] finished
0806 03:51:46.961521 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:46.961729 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101955,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":34026,"thread_start_us":435599,"threads_started":1,"wal-append.queue_time_us":507644}
I20260806 03:51:47.510018 4959 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:47.626502 2581 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 1.
I20260806 03:51:47.636502 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:47.654057 4743 raft_consensus.cc:3032] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:47.605358 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:47.659219 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1721) took 1194 ms. Trace:
I20260806 03:51:47.690703 2478 rpcz_store.cc:276] 0806 03:51:46.465271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.657738 (+1192467us) service_pool.cc:227] Handling call
0806 03:51:47.659154 (+ 1416us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.691733 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:47.613443 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:47.516476 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 720981us
W20260806 03:51:47.660681 2476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1707) took 2864 ms (client timeout 2104 ms). Trace:
W20260806 03:51:47.694037 2476 rpcz_store.cc:269] 0806 03:51:44.791196 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:44.947971 (+156775us) service_pool.cc:227] Handling call
0806 03:51:47.655283 (+2707312us) spinlock_profiling.cc:243] Waited 462 ms on lock 0x7b54002a0dd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:47.655443 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1062345088}
I20260806 03:51:47.613219 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:47.673689 2377 pending_rounds.cc:97] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 03:51:47.674214 2376 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:47.672680 2474 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:47.675225 5018 consensus_queue.cc:260] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:47.676676 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 154048us
W20260806 03:51:47.768675 2332 leader_election.cc:334] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.830s (SENT)
I20260806 03:51:47.769282 2332 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:47.677731 2577 raft_consensus.cc:1226] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260806 03:51:47.679849 5141 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.437s user 0.017s sys 0.009s
W20260806 03:51:47.681007 5199 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260806 03:51:47.656313 5116 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:47.696568 2375 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:47.688658 5127 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:47.693538 3542 leader_election.cc:334] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.030s (SENT)
I20260806 03:51:47.736784 2377 raft_consensus.cc:2866] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:47.737918 2370 raft_consensus.cc:2379] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:47.847760 2372 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260806 03:51:47.761482 2483 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1718) took 1472 ms (client timeout 1917 ms). Trace:
W20260806 03:51:47.848730 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 340297us
I20260806 03:51:47.611851 5156 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:47.857987 2537 connection.cc:232] RPC call timeout handler was delayed by -1.8147007520310581s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21810000000000002s for a response
I20260806 03:51:47.776258 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:47.858614 2537 connection.cc:232] RPC call timeout handler was delayed by -1.21820446010679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
I20260806 03:51:47.858954 2476 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:47.858963 2537 connection.cc:232] RPC call timeout handler was delayed by -1.1365409309510142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
I20260806 03:51:47.632418 4890 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:46.281232 2578 rpcz_store.cc:276] 0806 03:51:45.150663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.627061 (+476398us) service_pool.cc:227] Handling call
0806 03:51:45.752519 (+125458us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:45.752779 (+ 260us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:51:45.992879 (+240100us) log.cc:842] Serialized 14 byte log entry
0806 03:51:46.276598 (+283719us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:46.276665 (+ 67us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:46.276734 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:46.276975 (+ 241us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:46.277143 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4009,"thread_start_us":469110,"threads_started":2}
I20260806 03:51:47.808926 5087 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:47.799546 2373 raft_consensus.cc:2396] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:47.690439 4743 raft_consensus.cc:729] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:47.835705 2374 raft_consensus.cc:2396] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:47.840273 4959 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:47.838757 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 70149.3us
W20260806 03:51:47.891467 2332 leader_election.cc:334] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.920s (SENT)
I20260806 03:51:47.892127 2332 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:51:47.848762 2483 rpcz_store.cc:269] 0806 03:51:46.288693 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.522496 (+1233803us) service_pool.cc:227] Handling call
0806 03:51:47.761371 (+238875us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.851591 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1685) took 1002 ms. Trace:
I20260806 03:51:47.893558 2374 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
W20260806 03:51:47.852443 5067 consensus_meta.cc:287] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.789s user 0.015s sys 0.043s
I20260806 03:51:47.844825 5060 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:47.775400 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:47.856704 5018 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:47.863035 2433 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54220 dropped due to backpressure. The service queue is full; it has 50 items.
W20260806 03:51:47.696571 2575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 968) took 2064 ms (client timeout 1894 ms). Trace:
W20260806 03:51:47.844087 5004 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.010s sys 0.009s
W20260806 03:51:47.913036 2575 rpcz_store.cc:269] 0806 03:51:45.632231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.646789 (+ 14558us) service_pool.cc:227] Handling call
0806 03:51:47.696319 (+2049530us) spinlock_profiling.cc:243] Waited 233 ms on lock 0x7b5400454cd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:47.696483 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":536215296}
I20260806 03:51:47.914570 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:47.885947 2370 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:47.889237 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:47.888209 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1689) took 1037 ms. Trace:
I20260806 03:51:47.917402 2373 rpcz_store.cc:276] 0806 03:51:46.850708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.887170 (+1036462us) service_pool.cc:227] Handling call
0806 03:51:47.887486 (+ 316us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:47.887751 (+ 265us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:47.887788 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:47.887860 (+ 72us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:47.887909 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.887979 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:47.888129 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.893862 2372 rpcz_store.cc:276] 0806 03:51:46.849497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.786054 (+936557us) service_pool.cc:227] Handling call
0806 03:51:47.847481 (+ 61427us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:47.851179 (+ 3698us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.851524 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.852082 3542 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:47.869397 3586 raft_consensus.cc:2396] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:47.900321 2581 raft_consensus.cc:1226] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:47.901566 2480 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
W20260806 03:51:47.777640 2474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1969) took 1526 ms (client timeout 1991 ms). Trace:
I20260806 03:51:47.904771 2479 raft_consensus.cc:2396] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:51:47.905229 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 232311us
I20260806 03:51:47.927413 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:47.928344 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1722) took 1455 ms. Trace:
W20260806 03:51:47.928920 3547 leader_election.cc:334] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.269s (SENT)
I20260806 03:51:47.907387 2578 raft_consensus.cc:1226] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: []
I20260806 03:51:47.857124 5033 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:47.922135 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 228635us
I20260806 03:51:47.863091 5093 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:47.893738 4960 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:47.786420 5141 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:47.910910 2332 leader_election.cc:334] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.256s (SENT)
W20260806 03:51:47.957226 2479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1965) took 1725 ms (client timeout 2026 ms). Trace:
W20260806 03:51:47.957593 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 46731.9us
W20260806 03:51:47.957926 2479 rpcz_store.cc:269] 0806 03:51:46.238051 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.691395 (+1453344us) service_pool.cc:227] Handling call
0806 03:51:47.957112 (+265717us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.928696 2483 rpcz_store.cc:276] 0806 03:51:46.473286 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.893478 (+1420192us) service_pool.cc:227] Handling call
0806 03:51:47.928266 (+ 34788us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.890594 4993 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:47.926729 5004 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.011s sys 0.009s
W20260806 03:51:47.958202 2332 connection.cc:232] RPC call timeout handler was delayed by -1.5459181219339371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2397s for a response
I20260806 03:51:47.845520 5127 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:47.846835 5116 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:47.963457 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:47.966799 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:47.968900 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1691) took 1117 ms. Trace:
I20260806 03:51:47.978479 2372 rpcz_store.cc:276] 0806 03:51:46.851330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.937157 (+1085827us) service_pool.cc:227] Handling call
0806 03:51:47.968381 (+ 31224us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:47.968578 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:47.968605 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:47.968649 (+ 44us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:51:47.968680 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.968729 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:47.968838 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:47.979589 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.017s sys 0.027s
W20260806 03:51:47.979975 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.562s user 0.017s sys 0.027s
W20260806 03:51:47.980276 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 562092us
I20260806 03:51:47.969125 2373 raft_consensus.cc:1226] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:47.982317 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1690) took 1131 ms. Trace:
I20260806 03:51:47.982798 2373 rpcz_store.cc:276] 0806 03:51:46.851005 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.942086 (+1091081us) service_pool.cc:227] Handling call
0806 03:51:47.968938 (+ 26852us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:47.981998 (+ 13060us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.982234 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:47.891503 5209 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:47.984480 5209 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:47.970284 2332 connection.cc:232] RPC call timeout handler was delayed by -1.3015703312121332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25710000000000005s for a response
I20260806 03:51:47.912753 5087 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:47.922984 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:47.975241 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:47.988978 2578 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:47.989109 2373 raft_consensus.cc:1226] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260806 03:51:47.987272 2332 connection.cc:232] RPC call timeout handler was delayed by -1.1506923281122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17800000000000002s for a response
I20260806 03:51:47.991276 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1693) took 1139 ms. Trace:
I20260806 03:51:47.992272 2373 rpcz_store.cc:276] 0806 03:51:46.851965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.983742 (+1131777us) service_pool.cc:227] Handling call
0806 03:51:47.988219 (+ 4477us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:47.990495 (+ 2276us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.991061 (+ 566us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:47.991529 2332 connection.cc:232] RPC call timeout handler was delayed by -1.1326923279557377s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20260806 03:51:47.994763 2332 connection.cc:232] RPC call timeout handler was delayed by -1.030092328088358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19140000000000001s for a response
I20260806 03:51:47.936637 5134 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:48.002034 2484 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 03:51:48.002247 2476 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:51:48.002089 2332 leader_election.cc:334] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.780s (SENT)
W20260806 03:51:48.003507 2476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1724) took 1507 ms (client timeout 1863 ms). Trace:
I20260806 03:51:48.004055 2332 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:48.004102 2476 rpcz_store.cc:269] 0806 03:51:46.495882 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.695080 (+1199198us) service_pool.cc:227] Handling call
0806 03:51:48.003420 (+308340us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:47.975833 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.031s sys 0.031s
W20260806 03:51:48.005607 2332 leader_election.cc:334] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.800s (SENT)
W20260806 03:51:48.006105 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.812s user 0.032s sys 0.032s
I20260806 03:51:48.006243 2332 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:48.008524 2379 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 03:51:47.999125 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:48.010236 2581 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:47.845830 5130 consensus_queue.cc:260] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:48.019467 5130 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:48.020865 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:48.024647 2534 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:48.031857 2332 leader_election.cc:334] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.914s (SENT)
I20260806 03:51:48.032435 2332 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:48.032958 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:47.970899 5102 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:47.935783 2474 rpcz_store.cc:269] 0806 03:51:46.251307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.647207 (+1395900us) service_pool.cc:227] Handling call
0806 03:51:47.762295 (+115088us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:48.041045 2534 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:48.036011 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:48.036363 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:47.904639 5060 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:47.971907 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:48.043437 2534 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:48.057860 3542 leader_election.cc:334] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.836s (SENT)
I20260806 03:51:48.049280 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:48.058983 3542 leader_election.cc:334] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.983s (SENT)
I20260806 03:51:48.058084 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:48.059293 2373 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:48.058981 2534 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:48.039243 5023 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:48.020916 5087 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:48.020833 4797 raft_consensus.cc:3032] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:47.934015 5067 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:47.930119 4743 consensus_queue.cc:260] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:48.064415 2575 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
W20260806 03:51:48.076506 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 44719.4us
I20260806 03:51:48.076592 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:48.077247 2332 leader_election.cc:334] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.949s (SENT)
I20260806 03:51:48.021893 5154 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:48.031980 5035 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:48.020921 5075 consensus_queue.cc:1046] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260806 03:51:47.974364 4960 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:48.036196 4978 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:48.064342 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:48.090538 2577 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:48.077657 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:48.075253 2534 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:48.080124 2578 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:48.081393 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:47.965444 5093 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:47.906162 2436 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54220 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1984)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1978)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1966)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1981)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 130)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1988)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 133)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1989)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1990)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1991)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1992)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1993)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1994)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1995)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1986)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1997)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1998)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1987)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1711)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1957)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 131)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1963)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1970)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1971)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1972)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1975)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1976)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 134)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1977)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1982)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1983)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1726)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1728)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1729)
I20260806 03:51:48.078696 5204 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:48.087244 2537 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
W20260806 03:51:48.081663 5201 log.cc:924] Time spent T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.417s user 0.002s sys 0.000s
W20260806 03:51:48.104130 2369 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.604s user 0.008s sys 0.006s
W20260806 03:51:48.094314 2332 leader_election.cc:334] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.991s (SENT)
I20260806 03:51:48.067827 4797 raft_consensus.cc:729] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:48.095139 2481 raft_consensus.cc:2354] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:48.032984 4995 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:48.095794 2373 raft_consensus.cc:2396] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:48.088336 4743 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:48.111002 5211 log.cc:924] Time spent T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20260806 03:51:48.111912 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:48.112437 4890 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:48.112629 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1677) took 1467 ms. Trace:
I20260806 03:51:47.959371 5033 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:48.113252 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:48.072597 4856 raft_consensus.cc:3032] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:48.114203 4856 raft_consensus.cc:729] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:48.114753 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:48.115383 4856 consensus_queue.cc:260] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:48.056938 5016 consensus_queue.cc:1046] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 03:51:48.104877 2369 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
W20260806 03:51:48.105509 2332 leader_election.cc:334] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.000s (SENT)
I20260806 03:51:48.107162 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:48.108809 2575 raft_consensus.cc:2379] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:48.123466 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 165) took 1061 ms. Trace:
I20260806 03:51:48.113020 2368 rpcz_store.cc:276] 0806 03:51:46.645416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:46.646656 (+ 1240us) service_pool.cc:227] Handling call
0806 03:51:46.647136 (+ 480us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:46.852526 (+205390us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:46.852575 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:47.423640 (+571065us) log.cc:842] Serialized 26 byte log entry
0806 03:51:47.534676 (+111036us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:47.534741 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.534878 (+ 137us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:48.112379 (+577501us) raft_consensus.cc:1671] finished
0806 03:51:48.112401 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:48.112564 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":110486,"threads_started":1,"wal-append.queue_time_us":358889}
I20260806 03:51:48.118418 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:48.123602 2375 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 03:51:48.084059 5087 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:48.124356 2373 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:48.124465 2369 rpcz_store.cc:276] 0806 03:51:47.061829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.427834 (+366005us) service_pool.cc:227] Handling call
0806 03:51:48.123373 (+695539us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:48.125013 2332 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:48.068007 5023 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:48.097821 5035 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:48.149341 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:48.154942 2368 raft_consensus.cc:3032] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:48.155512 2368 raft_consensus.cc:729] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:48.092906 2578 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:51:48.157786 2578 raft_consensus.cc:595] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:48.158186 2578 raft_consensus.cc:603] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:51:48.158353 5093 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:48.159658 2578 raft_consensus.cc:1226] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3. Ops: []
W20260806 03:51:48.121745 5019 consensus_meta.cc:287] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 2.046s user 0.023s sys 0.007s
I20260806 03:51:48.159667 5116 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:48.127395 5193 log.cc:924] Time spent T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.286s user 0.000s sys 0.001s
W20260806 03:51:48.163975 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.550s user 0.003s sys 0.006s
I20260806 03:51:48.169661 2577 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:48.173270 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.559s user 0.003s sys 0.006s
I20260806 03:51:48.165109 2373 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
W20260806 03:51:48.163821 5202 log.cc:924] Time spent T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.325s user 0.000s sys 0.002s
I20260806 03:51:48.172222 2579 raft_consensus.cc:2396] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:48.165940 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1684) took 1316 ms. Trace:
I20260806 03:51:48.177145 2373 raft_consensus.cc:3032] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:48.099660 4978 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:48.166824 2368 consensus_queue.cc:260] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:48.160285 5035 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:48.163112 5087 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:51:48.165205 4856 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:48.190069 2376 rpcz_store.cc:276] 0806 03:51:46.849184 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.737461 (+888277us) service_pool.cc:227] Handling call
0806 03:51:47.783691 (+ 46230us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:47.783930 (+ 239us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:48.022718 (+238788us) log.cc:842] Serialized 14 byte log entry
0806 03:51:48.165550 (+142832us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:48.165610 (+ 60us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:48.165655 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.165721 (+ 66us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:48.165853 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":342920,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":109907,"thread_start_us":319708,"threads_started":3}
W20260806 03:51:48.213670 2331 connection.cc:232] RPC call timeout handler was delayed by -4.38488839706406s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
I20260806 03:51:48.223369 2575 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:51:48.223889 2368 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:48.233887 2331 connection.cc:232] RPC call timeout handler was delayed by -4.3659883970394731s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
W20260806 03:51:48.234462 2331 connection.cc:232] RPC call timeout handler was delayed by -4.0923883970826864s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1854s for a response
W20260806 03:51:48.234921 2331 connection.cc:232] RPC call timeout handler was delayed by -4.03118839696981s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
W20260806 03:51:48.235371 2331 connection.cc:232] RPC call timeout handler was delayed by -3.7881883969530463s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21920000000000003s for a response
I20260806 03:51:48.222652 5111 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:48.224609 5087 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:48.236438 4960 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:48.236585 5111 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:48.224135 5035 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:48.233062 2538 leader_election.cc:334] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.785s (SENT)
W20260806 03:51:48.235795 2331 connection.cc:232] RPC call timeout handler was delayed by -3.7305883970111609s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2256s for a response
I20260806 03:51:48.238967 2538 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:48.239459 2331 connection.cc:232] RPC call timeout handler was delayed by -3.3381883969996125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26920000000000005s for a response
I20260806 03:51:48.244738 2331 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:48.230628 2373 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:48.218915 2436 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:56244 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1981)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 130)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1988)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 133)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1989)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1990)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1991)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1992)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1993)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1994)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1995)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1986)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1997)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1998)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1987)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1999)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2000)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2001)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1711)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1957)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 131)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1963)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1970)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1971)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1972)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1975)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1976)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 134)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1977)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1982)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1983)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1726)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1728)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1729)
I20260806 03:51:48.229789 5023 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:48.246479 2373 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:48.261199 2373 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:48.231698 4745 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:48.258352 5111 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:48.256680 5087 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:48.181999 5016 raft_consensus.cc:2781] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election won for term 2
I20260806 03:51:48.267568 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:48.264827 2581 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:48.276101 2376 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
W20260806 03:51:48.225468 5119 consensus_meta.cc:287] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 2.165s user 0.017s sys 0.011s
W20260806 03:51:48.278029 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 180495us
I20260806 03:51:48.176124 5216 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:48.192374 5203 log.cc:924] Time spent T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.292s user 0.000s sys 0.002s
I20260806 03:51:48.308442 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:48.299031 2577 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:48.288882 2436 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54234 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1981)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 130)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1988)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 133)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1989)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1990)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1991)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1992)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1993)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1994)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1995)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1986)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1997)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1998)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1987)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1999)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2000)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2001)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1711)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1957)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 131)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1963)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1970)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1971)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1972)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1975)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1976)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 134)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1977)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1982)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1983)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1726)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1728)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1729)
I20260806 03:51:48.298307 2368 raft_consensus.cc:1261] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:48.215976 5236 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:48.225423 5060 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:48.216087 5156 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:48.306483 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 200993us
W20260806 03:51:48.307173 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 220023us
I20260806 03:51:48.222795 4978 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:48.230194 5093 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:48.317058 2373 raft_consensus.cc:1261] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 03:51:48.426823 2575 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:48.414868 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1686) took 1565 ms. Trace:
I20260806 03:51:48.428411 2583 raft_consensus.cc:1226] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:48.484916 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1692) took 1633 ms. Trace:
I20260806 03:51:48.410599 2578 raft_consensus.cc:1226] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260806 03:51:48.261950 5221 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:51:48.489254 2436 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1729) took 1583 ms. Trace:
W20260806 03:51:48.449868 2480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1715) took 2171 ms (client timeout 1901 ms). Trace:
I20260806 03:51:48.275404 4745 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:48.420730 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 978) took 1843 ms. Trace:
I20260806 03:51:48.540319 2575 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:48.939695 2579 rpcz_store.cc:276] 0806 03:51:46.577435 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:46.731208 (+153773us) service_pool.cc:227] Handling call
0806 03:51:48.420483 (+1689275us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b5400156154. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:48.420651 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3310377344}
I20260806 03:51:48.940542 2581 raft_consensus.cc:1226] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: []
W20260806 03:51:48.415402 2482 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1964) took 2264 ms (client timeout 1949 ms). Trace:
W20260806 03:51:48.567129 3591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 325) took 2964 ms (client timeout 1944 ms). Trace:
I20260806 03:51:48.490074 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1702) took 1067 ms. Trace:
I20260806 03:51:48.457569 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:48.599241 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 144576us
W20260806 03:51:48.489359 2332 leader_election.cc:334] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.026s (SENT)
W20260806 03:51:48.573484 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 265105us
W20260806 03:51:48.339076 2576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 964) took 2958 ms (client timeout 2851 ms). Trace:
I20260806 03:51:48.944841 2332 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:48.413831 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1688) took 1563 ms. Trace:
I20260806 03:51:48.428179 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1695) took 1254 ms. Trace:
W20260806 03:51:48.538931 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 47541.8us
I20260806 03:51:48.493443 2373 pending_rounds.cc:76] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 1
I20260806 03:51:48.945799 2374 rpcz_store.cc:276] 0806 03:51:46.850401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.885140 (+1034739us) service_pool.cc:227] Handling call
0806 03:51:47.885558 (+ 418us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:47.894135 (+ 8577us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.413506 (+519371us) spinlock_profiling.cc:243] Waited 7.72 ms on lock 0x7b5400059bd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:48.413751 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17760128}
I20260806 03:51:48.946965 2373 pending_rounds.cc:97] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 2.2
W20260806 03:51:48.854638 3542 leader_election.cc:334] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.269s (SENT)
I20260806 03:51:48.950572 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2062) took 1015 ms. Trace:
I20260806 03:51:48.946475 2369 rpcz_store.cc:276] 0806 03:51:47.173686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.125739 (+952053us) service_pool.cc:227] Handling call
0806 03:51:48.126421 (+ 682us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:48.161003 (+ 34582us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:48.161047 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:48.166223 (+ 5176us) log.cc:842] Serialized 26 byte log entry
0806 03:51:48.352900 (+186677us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:48.352964 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.353046 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:48.427849 (+ 74803us) raft_consensus.cc:1671] finished
0806 03:51:48.427880 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:48.428092 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4315,"raft.queue_time_us":80451,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":127,"thread_start_us":128992,"threads_started":1,"wal-append.queue_time_us":240219}
I20260806 03:51:48.948813 2483 raft_consensus.cc:2379] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:51:48.952807 2483 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1966) took 2702 ms (client timeout 2386 ms). Trace:
W20260806 03:51:48.953233 2483 rpcz_store.cc:269] 0806 03:51:46.250307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.457166 (+2206859us) service_pool.cc:227] Handling call
0806 03:51:48.952721 (+495555us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:48.954239 2483 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 130) took 2852 ms (client timeout 2797 ms). Trace:
I20260806 03:51:48.452047 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:48.956365 2474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1725) took 2331 ms (client timeout 2103 ms). Trace:
W20260806 03:51:48.956825 2474 rpcz_store.cc:269] 0806 03:51:46.624685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.451543 (+1826858us) service_pool.cc:227] Handling call
0806 03:51:48.956264 (+504721us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:48.957985 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:48.958781 2474 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:48.347316 5224 log.cc:924] Time spent T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
I20260806 03:51:48.572240 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1701) took 4270 ms. Trace:
W20260806 03:51:48.941447 2482 rpcz_store.cc:269] 0806 03:51:46.150726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:46.213949 (+ 63223us) service_pool.cc:227] Handling call
0806 03:51:48.415213 (+2201264us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54002c33d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:48.415319 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":417731328}
W20260806 03:51:48.598855 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 222103us
W20260806 03:51:48.412431 5231 log.cc:924] Time spent T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.314s user 0.000s sys 0.001s
I20260806 03:51:48.950603 2373 raft_consensus.cc:2866] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:48.746985 4797 consensus_queue.cc:260] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:48.970059 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1681) took 2122 ms. Trace:
I20260806 03:51:48.970592 4797 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:48.970427 2377 rpcz_store.cc:276] 0806 03:51:46.848193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.670524 (+822331us) service_pool.cc:227] Handling call
0806 03:51:47.670947 (+ 423us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:47.847024 (+176077us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:47.847066 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:47.847579 (+ 513us) log.cc:842] Serialized 26 byte log entry
0806 03:51:48.002091 (+154512us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:48.002144 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.003022 (+ 878us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:48.969796 (+966774us) raft_consensus.cc:1671] finished
0806 03:51:48.969820 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:48.969995 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":102763,"threads_started":1,"wal-append.queue_time_us":205516}
I20260806 03:51:48.942638 2436 rpcz_store.cc:276] 0806 03:51:46.905790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.489194 (+1583404us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20260806 03:51:48.943296 2480 rpcz_store.cc:269] 0806 03:51:46.278126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.507488 (+1229362us) service_pool.cc:227] Handling call
0806 03:51:48.449601 (+942113us) spinlock_profiling.cc:243] Waited 278 ms on lock 0x7b54000afad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:48.449768 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":640029056}
I20260806 03:51:48.491663 2375 rpcz_store.cc:276] 0806 03:51:46.849789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.786468 (+936679us) service_pool.cc:227] Handling call
0806 03:51:48.123330 (+336862us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:48.128337 (+ 5007us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.414652 (+286315us) spinlock_profiling.cc:243] Waited 336 ms on lock 0x7b5400007650. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:48.414793 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":773715328}
I20260806 03:51:48.951814 2374 raft_consensus.cc:1226] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:51:48.279111 5001 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:48.279839 5209 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:48.308604 5141 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:48.413056 5222 log.cc:924] Time spent T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.329s user 0.001s sys 0.001s
I20260806 03:51:48.377278 5067 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:48.564690 5236 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:48.975893 5060 consensus_queue.cc:1046] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:48.458576 5035 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:48.976646 5221 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:51:48.640463 5204 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.524s user 0.012s sys 0.000s
I20260806 03:51:48.948048 5093 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:48.949105 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 459767us
I20260806 03:51:48.952255 2583 rpcz_store.cc:276] 0806 03:51:47.935035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.365864 (+430829us) service_pool.cc:227] Handling call
0806 03:51:48.411278 (+ 45414us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:48.946805 (+535527us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.950505 (+ 3700us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:48.983788 2332 leader_election.cc:334] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.039s (SENT)
W20260806 03:51:48.941823 3591 rpcz_store.cc:269] 0806 03:51:45.602994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.645581 (+ 42587us) service_pool.cc:227] Handling call
0806 03:51:48.566844 (+2921263us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b54005fe954. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:48.566999 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":741479168}
I20260806 03:51:48.943857 2368 rpcz_store.cc:276] 0806 03:51:47.422182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.154319 (+732137us) service_pool.cc:227] Handling call
0806 03:51:48.154769 (+ 450us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:48.489739 (+334970us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.490004 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:48.942852 2372 rpcz_store.cc:276] 0806 03:51:46.851680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.979813 (+1128133us) service_pool.cc:227] Handling call
0806 03:51:47.980205 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:47.980578 (+ 373us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:51:48.084697 (+104119us) log.cc:842] Serialized 14 byte log entry
0806 03:51:48.484420 (+399723us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:48.484493 (+ 73us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:48.484563 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.484661 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:48.484809 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":176047,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":6057,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":80932,"spinlock_wait_cycles":2963328,"thread_start_us":447154,"threads_started":2,"wal-append.queue_time_us":489537}
W20260806 03:51:48.974536 5130 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.941s user 0.013s sys 0.003s
W20260806 03:51:48.963017 2483 rpcz_store.cc:269] 0806 03:51:46.101492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.954061 (+2852569us) service_pool.cc:214] Skipping call since client already timed out
0806 03:51:48.954166 (+ 105us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 03:51:48.945477 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:48.306787 5119 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:48.736256 5018 consensus_meta.cc:287] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.796s user 0.010s sys 0.004s
W20260806 03:51:48.729485 5007 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.663s user 0.029s sys 0.035s
W20260806 03:51:49.005128 5007 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.938s user 0.040s sys 0.041s
I20260806 03:51:49.005952 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 979) took 2123 ms. Trace:
I20260806 03:51:49.006455 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:49.006754 5119 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:49.007031 3590 raft_consensus.cc:1226] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:49.006386 2574 rpcz_store.cc:276] 0806 03:51:46.882492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.641902 (+759410us) service_pool.cc:227] Handling call
0806 03:51:47.642430 (+ 528us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:47.642747 (+ 317us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:47.642783 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:47.643403 (+ 620us) log.cc:842] Serialized 26 byte log entry
0806 03:51:47.963236 (+319833us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:47.963293 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:47.963403 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:49.005697 (+1042294us) raft_consensus.cc:1671] finished
0806 03:51:49.005721 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:49.005871 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":319325,"threads_started":1,"wal-append.queue_time_us":449021}
I20260806 03:51:49.010628 3590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36468 (request call id 330) took 1581 ms. Trace:
I20260806 03:51:49.011101 3590 rpcz_store.cc:276] 0806 03:51:47.428983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.429152 (+ 169us) service_pool.cc:227] Handling call
0806 03:51:47.429457 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.007457 (+1578000us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.010349 (+ 2892us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400540454. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:49.010557 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3627580160}
I20260806 03:51:49.012190 5035 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:48.945334 2576 rpcz_store.cc:269] 0806 03:51:45.381075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.623870 (+242795us) service_pool.cc:227] Handling call
0806 03:51:48.338771 (+2714901us) spinlock_profiling.cc:243] Waited 596 ms on lock 0x7b54004466d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:48.338965 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1370772736}
W20260806 03:51:48.655256 5168 log.cc:924] Time spent T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 2.207s user 0.004s sys 0.025s
I20260806 03:51:48.965780 2484 rpcz_store.cc:276] 0806 03:51:44.302256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:46.085295 (+1783039us) service_pool.cc:227] Handling call
0806 03:51:46.085666 (+ 371us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:48.002532 (+1916866us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.571864 (+569332us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b54001dced4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:48.572160 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4406697728}
I20260806 03:51:48.421192 5033 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:48.968734 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1703) took 1546 ms. Trace:
I20260806 03:51:48.963282 5247 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:49.013427 3591 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:51:48.969336 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:48.327055 5087 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:48.950913 3542 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:48.743209 5246 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:48.327348 5016 raft_consensus.cc:686] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:49.029209 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 174652us
I20260806 03:51:49.030010 5221 consensus_queue.cc:1046] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:51:49.030191 5016 consensus_queue.cc:237] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:49.031131 4983 raft_consensus.cc:3032] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:51:49.031523 4983 raft_consensus.cc:729] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:49.032048 3542 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:49.032965 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:49.033769 3586 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:49.035386 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:49.036194 2484 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1991) took 1431 ms (client timeout 1800 ms). Trace:
I20260806 03:51:49.036649 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:49.036558 2484 rpcz_store.cc:269] 0806 03:51:47.605811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.017135 (+1411324us) service_pool.cc:227] Handling call
0806 03:51:49.036129 (+ 18994us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:48.508038 5216 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:48.568639 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:48.972677 2436 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54220 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1990)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1991)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1992)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1993)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1994)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1995)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2002)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1986)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1997)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2003)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1998)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1987)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2005)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1999)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2000)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2001)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1711)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1957)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 131)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1963)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1970)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1971)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1972)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1975)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1976)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 134)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1977)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1982)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1983)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1726)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1728)
I20260806 03:51:48.982115 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.046480 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.046844 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:49.047030 2480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1989) took 1527 ms (client timeout 1749 ms). Trace:
I20260806 03:51:49.047704 2484 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:48.309792 5023 consensus_queue.cc:1046] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.665s
I20260806 03:51:48.984416 2332 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:49.047461 2480 rpcz_store.cc:269] 0806 03:51:47.520133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.973310 (+1453177us) service_pool.cc:227] Handling call
0806 03:51:49.046925 (+ 73615us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:48.978346 4856 consensus_meta.cc:287] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.753s user 0.007s sys 0.006s
W20260806 03:51:48.419637 4906 consensus_meta.cc:287] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.896s user 0.011s sys 0.000s
I20260806 03:51:48.989267 5060 raft_consensus.cc:3032] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
W20260806 03:51:48.993690 5234 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.778s user 0.006s sys 0.001s
I20260806 03:51:48.985900 5067 consensus_queue.cc:1046] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260806 03:51:48.995230 5236 raft_consensus.cc:2741] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:51:48.983253 5156 consensus_queue.cc:1046] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260806 03:51:48.996749 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:48.998072 2583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:51:49.064612 2583 raft_consensus.cc:595] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:48.998562 2481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1974) took 2745 ms (client timeout 1920 ms). Trace:
I20260806 03:51:49.064903 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:49.065218 2481 rpcz_store.cc:269] 0806 03:51:46.261929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.506674 (+1244745us) service_pool.cc:227] Handling call
0806 03:51:48.998329 (+1491655us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54003886d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:48.998473 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1115016448}
I20260806 03:51:49.001807 2478 raft_consensus.cc:2396] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:49.002508 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:48.977540 4978 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260806 03:51:49.024652 5111 consensus_queue.cc:1046] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 03:51:49.013929 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.013309 5141 consensus_queue.cc:1046] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260806 03:51:49.018378 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.016288 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:49.015829 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.019230 2373 rpcz_store.cc:276] 0806 03:51:47.422554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.175000 (+752446us) service_pool.cc:227] Handling call
0806 03:51:48.176968 (+ 1968us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:48.968438 (+791470us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:48.968662 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:48.984364 5204 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:49.021699 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:49.072576 2377 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:49.075187 2368 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:49.025729 2482 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 133) took 2911 ms (client timeout 3092 ms). Trace:
I20260806 03:51:49.024844 5163 consensus_queue.cc:1046] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.002s
I20260806 03:51:49.026638 5246 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:49.028525 5113 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:49.028837 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:49.086499 4986 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:49.087623 2373 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:49.091035 5018 raft_consensus.cc:3032] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:49.091429 5018 raft_consensus.cc:729] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:49.093355 5156 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:49.093181 4856 consensus_queue.cc:1035] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260806 03:51:49.093860 5156 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:49.095737 5230 log.cc:924] Time spent T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.881s user 0.000s sys 0.003s
I20260806 03:51:49.096007 5156 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:49.083595 2368 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:49.038247 2537 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:49.038511 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:49.037835 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:49.107532 2575 raft_consensus.cc:2362] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4. Re-sending same reply.
I20260806 03:51:49.073004 5141 raft_consensus.cc:3032] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:51:49.049131 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 65362.8us
W20260806 03:51:49.110842 2332 leader_election.cc:334] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.192s (SENT)
I20260806 03:51:49.048177 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1992) took 1431 ms. Trace:
I20260806 03:51:49.111364 2332 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:49.111753 2484 rpcz_store.cc:276] 0806 03:51:47.617084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.038457 (+1421373us) service_pool.cc:227] Handling call
0806 03:51:49.048109 (+ 9652us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:49.113040 2332 leader_election.cc:334] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.386s (SENT)
I20260806 03:51:49.115124 2332 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:49.115590 2534 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:49.022660 5119 consensus_queue.cc:1046] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.030s
W20260806 03:51:49.120769 2332 leader_election.cc:334] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.397s (SENT)
I20260806 03:51:49.121369 2332 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:49.122771 2332 leader_election.cc:334] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.571s (SENT)
I20260806 03:51:49.123216 2332 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:49.060738 3588 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:49.065712 2480 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:49.066191 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.066397 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.125298 2481 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
W20260806 03:51:49.125927 2481 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1993) took 1464 ms (client timeout 1949 ms). Trace:
W20260806 03:51:49.126344 2481 rpcz_store.cc:269] 0806 03:51:47.662414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.066133 (+1403719us) service_pool.cc:227] Handling call
0806 03:51:49.125849 (+ 59716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.000559 5093 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:49.004454 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.056428 5060 raft_consensus.cc:729] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:49.110253 5141 raft_consensus.cc:729] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:49.134651 2372 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:49.135216 5060 consensus_queue.cc:260] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:49.136749 5060 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:49.137507 5119 raft_consensus.cc:3032] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:49.138091 5119 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:49.051847 4993 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:49.140367 4981 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:49.140519 4993 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:49.078564 2482 rpcz_store.cc:269] 0806 03:51:46.113928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.966285 (+2852357us) service_pool.cc:227] Handling call
0806 03:51:49.025654 (+ 59369us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.141117 4981 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:49.141825 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.073766 3590 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:49.142752 2482 raft_consensus.cc:2396] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:51:49.073971 2483 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1990) took 1514 ms (client timeout 1780 ms). Trace:
W20260806 03:51:49.143610 2483 rpcz_store.cc:269] 0806 03:51:47.560167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.986247 (+1426080us) service_pool.cc:227] Handling call
0806 03:51:49.073868 (+ 87621us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.145123 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:49.080896 2377 raft_consensus.cc:3032] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:49.146588 2377 raft_consensus.cc:729] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:49.046217 5019 consensus_queue.cc:1035] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260806 03:51:49.148443 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:49.149261 3585 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:49.149542 5019 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:49.150256 5019 raft_consensus.cc:686] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:49.064198 4906 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:49.151889 5018 consensus_queue.cc:260] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:49.071321 2477 raft_consensus.cc:2379] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:51:49.152951 2480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1716) took 2874 ms (client timeout 3272 ms). Trace:
W20260806 03:51:49.153412 2477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1981) took 2589 ms (client timeout 2256 ms). Trace:
I20260806 03:51:49.105890 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:49.106477 2476 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1984) took 2464 ms (client timeout 1830 ms). Trace:
I20260806 03:51:49.127483 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:49.154888 2476 rpcz_store.cc:269] 0806 03:51:46.642386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.106217 (+2463831us) service_pool.cc:214] Skipping call since client already timed out
0806 03:51:49.106381 (+ 164us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 03:51:49.039505 2576 raft_consensus.cc:2379] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:49.042415 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.044746 4983 consensus_queue.cc:260] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:49.155699 2481 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:49.160597 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1995) took 1471 ms. Trace:
W20260806 03:51:49.109730 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 71615.4us
I20260806 03:51:49.082404 4978 consensus_queue.cc:1225] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer cf6042e85bef47c4aac9d064d91501d1 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 03:51:49.156543 2574 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:49.119745 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:49.129197 5087 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:49.129623 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:49.166728 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:49.129561 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.172668 3590 raft_consensus.cc:2362] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3. Re-sending same reply.
I20260806 03:51:49.172837 2484 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:49.147737 2377 consensus_queue.cc:260] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:49.173316 2484 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1994) took 1506 ms (client timeout 1991 ms). Trace:
I20260806 03:51:49.173663 2377 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:49.133526 5246 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:49.153411 2480 rpcz_store.cc:269] 0806 03:51:46.278547 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.048826 (+2770279us) service_pool.cc:227] Handling call
0806 03:51:49.152877 (+104051us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.093222 5008 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:49.161072 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.184041 2476 raft_consensus.cc:2379] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:49.184679 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1986) took 1750 ms. Trace:
I20260806 03:51:49.185070 2476 rpcz_store.cc:276] 0806 03:51:47.434477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.155876 (+1721399us) service_pool.cc:227] Handling call
0806 03:51:49.184590 (+ 28714us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.160993 2481 rpcz_store.cc:276] 0806 03:51:47.689732 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.127158 (+1437426us) service_pool.cc:227] Handling call
0806 03:51:49.160533 (+ 33375us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.186655 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.187753 2481 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:49.188303 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1998) took 1329 ms. Trace:
I20260806 03:51:49.188669 2481 rpcz_store.cc:276] 0806 03:51:47.858920 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.186275 (+1327355us) service_pool.cc:227] Handling call
0806 03:51:49.188233 (+ 1958us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:49.166198 2332 leader_election.cc:334] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.692s (SENT)
I20260806 03:51:49.189610 2332 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:49.168736 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:49.169019 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.171116 2537 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:49.173663 2484 rpcz_store.cc:269] 0806 03:51:47.667328 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.113161 (+1445833us) service_pool.cc:227] Handling call
0806 03:51:49.173257 (+ 60096us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.143210 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1996) took 1382 ms. Trace:
I20260806 03:51:49.196978 2482 rpcz_store.cc:276] 0806 03:51:47.760594 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.141482 (+1380888us) service_pool.cc:227] Handling call
0806 03:51:49.143148 (+ 1666us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.114933 5236 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:49.164708 3545 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:49.208168 2537 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:49.148361 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:49.117527 5113 raft_consensus.cc:3032] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:51:49.129084 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:49.225565 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:49.232640 2576 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260806 03:51:49.225208 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:49.229867 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 63691.5us
W20260806 03:51:49.154023 2477 rpcz_store.cc:269] 0806 03:51:46.563890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.568219 (+2004329us) service_pool.cc:227] Handling call
0806 03:51:49.153297 (+585078us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:49.224385 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 59766.5us
I20260806 03:51:49.241269 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:49.243834 2537 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:49.222054 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.245882 2482 raft_consensus.cc:2379] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:49.246616 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.237154 2579 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:49.247618 2480 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:49.222054 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.249086 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1987) took 1750 ms. Trace:
I20260806 03:51:49.249481 2481 rpcz_store.cc:276] 0806 03:51:47.498554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.221699 (+1723145us) service_pool.cc:227] Handling call
0806 03:51:49.248998 (+ 27299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":62976}
I20260806 03:51:49.223742 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.251116 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1997) took 1408 ms. Trace:
I20260806 03:51:49.154136 4856 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
W20260806 03:51:49.238238 2332 leader_election.cc:334] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.838s (SENT)
I20260806 03:51:49.160260 5221 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:49.235860 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.252313 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1999) took 1080 ms. Trace:
I20260806 03:51:49.258455 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:49.252322 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.258373 2482 rpcz_store.cc:276] 0806 03:51:48.172231 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.221618 (+1049387us) service_pool.cc:227] Handling call
0806 03:51:49.252239 (+ 30621us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41088}
I20260806 03:51:49.260004 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.236743 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:49.261427 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2003) took 1043 ms. Trace:
I20260806 03:51:49.261931 2476 rpcz_store.cc:276] 0806 03:51:48.218352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.185883 (+967531us) service_pool.cc:227] Handling call
0806 03:51:49.261343 (+ 75460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.257851 2332 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:49.253535 2480 rpcz_store.cc:276] 0806 03:51:47.843193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.178834 (+1335641us) service_pool.cc:227] Handling call
0806 03:51:49.251035 (+ 72201us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:49.238431 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 256337us
W20260806 03:51:49.293330 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 261487us
I20260806 03:51:49.294102 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:49.295527 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 179995us
I20260806 03:51:49.200210 4993 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:51:49.298079 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.151511 5019 consensus_queue.cc:237] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:49.139272 5119 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:49.229061 5093 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:49.316879 2372 raft_consensus.cc:3032] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:51:49.295634 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.317315 2372 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:49.223002 4978 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:49.334203 2572 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 03:51:49.135216 5141 consensus_queue.cc:260] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:49.307075 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.239279 5087 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:49.231671 5113 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:49.308286 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.341922 2476 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:49.311453 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:49.312485 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.343618 2576 raft_consensus.cc:2379] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:49.202348 5035 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
I20260806 03:51:49.255769 5256 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:49.325220 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.256752 4797 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:49.297865 5023 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:49.329530 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 160118us
I20260806 03:51:49.338915 5119 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:49.332013 2377 raft_consensus.cc:1261] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20260806 03:51:49.318471 5250 log.cc:924] Time spent T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260806 03:51:49.334623 2572 pending_rounds.cc:76] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 2
I20260806 03:51:49.359849 2572 pending_rounds.cc:97] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:51:49.360174 2572 raft_consensus.cc:2866] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:49.361104 5138 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:49.361625 5138 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:49.335732 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:49.221233 5008 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:49.326062 3586 raft_consensus.cc:2379] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 2: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:49.303921 2534 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:49.369729 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 65921.1us
I20260806 03:51:49.337663 2372 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:49.372056 2372 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:49.350535 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 974) took 2794 ms. Trace:
I20260806 03:51:49.329854 5093 raft_consensus.cc:686] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:49.351613 2484 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:49.333431 4978 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:49.354159 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.329244 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.303134 5204 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:49.364522 2377 pending_rounds.cc:76] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 1
I20260806 03:51:49.373231 2584 rpcz_store.cc:276] 0806 03:51:46.555965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.419828 (+863863us) service_pool.cc:227] Handling call
0806 03:51:47.420379 (+ 551us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:47.623513 (+203134us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:47.623560 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:47.624226 (+ 666us) log.cc:842] Serialized 26 byte log entry
0806 03:51:48.735190 (+1110964us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b18002bc120. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b7f345 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9ce22 00007f0d7c7465bb 00007f0d7c74c063 00007f0d7c74c411 00007f0d7ca995cf
0806 03:51:48.941845 (+206655us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:49.341363 (+399518us) log.cc:842] Serialized 14 byte log entry
0806 03:51:49.344461 (+ 3098us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.344605 (+ 144us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:49.344669 (+ 64us) raft_consensus.cc:1671] finished
0806 03:51:49.350277 (+ 5608us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:49.350460 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8039,"prepare.queue_time_us":314253,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":198520,"raft.run_cpu_time_us":576,"raft.run_wall_time_us":21275,"spinlock_wait_cycles":46076288,"thread_start_us":782647,"threads_started":3,"wal-append.queue_time_us":1333201}
I20260806 03:51:49.382941 2377 pending_rounds.cc:97] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260806 03:51:49.383383 2377 raft_consensus.cc:2866] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:49.347023 5102 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:49.357203 5141 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:49.325911 5268 consensus_queue.cc:1046] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 03:51:49.386669 2480 raft_consensus.cc:2396] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:49.397719 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2001) took 1180 ms. Trace:
I20260806 03:51:49.398218 2480 rpcz_store.cc:276] 0806 03:51:48.217653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.263539 (+1045886us) service_pool.cc:227] Handling call
0806 03:51:49.397638 (+134099us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.400720 2482 raft_consensus.cc:2396] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:51:49.402263 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.403138 2484 raft_consensus.cc:2379] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:49.385900 4978 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:49.404481 2484 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260806 03:51:49.409183 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.413964 2476 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:49.413048 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1703) took 5110 ms. Trace:
I20260806 03:51:49.415138 3547 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:49.417392 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.422307 2372 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260806 03:51:49.422680 2372 pending_rounds.cc:76] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 6
I20260806 03:51:49.422884 2372 pending_rounds.cc:97] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260806 03:51:49.423112 2372 raft_consensus.cc:2866] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:49.415227 2484 rpcz_store.cc:276] 0806 03:51:44.302788 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.403781 (+5100993us) service_pool.cc:227] Handling call
0806 03:51:49.404210 (+ 429us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:51:49.412735 (+ 8525us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.412986 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.418087 2377 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
W20260806 03:51:49.426211 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 182488us
I20260806 03:51:49.426640 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:49.427121 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:49.427580 2372 raft_consensus.cc:2362] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4. Re-sending same reply.
I20260806 03:51:49.427742 2377 raft_consensus.cc:2362] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4. Re-sending same reply.
I20260806 03:51:49.385706 5116 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:49.403961 2474 raft_consensus.cc:2396] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:51:49.380211 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 119198us
I20260806 03:51:49.432512 2476 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:49.358583 4797 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:49.433776 2484 raft_consensus.cc:1226] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:49.432777 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:49.432066 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.438562 2480 raft_consensus.cc:2379] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:49.439931 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:49.435225 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 128) took 5090 ms. Trace:
I20260806 03:51:49.443141 2476 rpcz_store.cc:276] 0806 03:51:44.344720 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.414656 (+5069936us) service_pool.cc:227] Handling call
0806 03:51:49.432153 (+ 17497us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.434925 (+ 2772us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.435159 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.416389 4906 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:49.367933 5023 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:49.408628 5204 raft_consensus.cc:686] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:49.449867 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1708) took 4657 ms. Trace:
I20260806 03:51:49.450747 2484 rpcz_store.cc:276] 0806 03:51:44.792307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.433277 (+4640970us) service_pool.cc:227] Handling call
0806 03:51:49.433587 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.446217 (+ 12630us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.449769 (+ 3552us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.407226 3585 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:51:49.431258 5116 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:49.425796 4986 consensus_queue.cc:1035] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260806 03:51:49.445387 5271 raft_consensus.cc:2781] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260806 03:51:49.452153 2474 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1988) took 1933 ms (client timeout 1597 ms). Trace:
I20260806 03:51:49.470010 2480 raft_consensus.cc:1261] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:49.434754 5278 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
I20260806 03:51:49.412985 5113 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:49.467372 5093 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:47.946655 2433 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:54220 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1981)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 130)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1988)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 133)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1989)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1990)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1991)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1992)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1716)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1993)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1994)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1995)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1996)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1986)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1997)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1998)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1987)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1702)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1703)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 128)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1708)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1710)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1711)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1957)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1713)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1714)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 129)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 131)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1963)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1970)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1971)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1972)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1975)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1976)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1720)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 134)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1723)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1977)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1982)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1983)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 135)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1726)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1727)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1728)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1729)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 136)
I20260806 03:51:49.472795 2484 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:51:49.457590 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:49.558109 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1687) took 2708 ms. Trace:
I20260806 03:51:49.663223 2370 rpcz_store.cc:276] 0806 03:51:46.850102 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.856533 (+1006431us) service_pool.cc:227] Handling call
0806 03:51:47.885749 (+ 29216us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:47.916359 (+ 30610us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.557835 (+1641476us) spinlock_profiling.cc:243] Waited 28.8 ms on lock 0x7b540005a5d0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:49.558032 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":66198400}
I20260806 03:51:49.662858 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1710) took 4869 ms. Trace:
I20260806 03:51:49.664580 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.664996 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1957) took 3901 ms. Trace:
W20260806 03:51:49.500669 2478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1985) took 2857 ms (client timeout 1673 ms). Trace:
I20260806 03:51:49.665571 2484 rpcz_store.cc:276] 0806 03:51:45.763580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.457233 (+3693653us) service_pool.cc:227] Handling call
0806 03:51:49.471148 (+ 13915us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.664719 (+193571us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.664931 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:49.666404 2478 rpcz_store.cc:269] 0806 03:51:46.843132 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:47.691442 (+848310us) service_pool.cc:227] Handling call
0806 03:51:49.479922 (+1788480us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54002a03d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:49.500561 (+ 20639us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3010573568}
I20260806 03:51:49.664354 2480 rpcz_store.cc:276] 0806 03:51:44.793037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.439712 (+4646675us) service_pool.cc:227] Handling call
0806 03:51:49.469117 (+ 29405us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.662475 (+193358us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.662768 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.581295 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.591548 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:44180 (request call id 400) took 1723 ms. Trace:
I20260806 03:51:49.490792 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 153) took 3230 ms. Trace:
I20260806 03:51:49.671181 2379 rpcz_store.cc:276] 0806 03:51:46.260461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:46.262355 (+ 1894us) service_pool.cc:227] Handling call
0806 03:51:46.283084 (+ 20729us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:48.016824 (+1733740us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.490543 (+1473719us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b54000c0654. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:49.490727 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3950273920}
W20260806 03:51:49.515844 2479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 1978) took 2997 ms (client timeout 2032 ms). Trace:
W20260806 03:51:49.672416 2479 rpcz_store.cc:269] 0806 03:51:46.518694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.515630 (+2996936us) service_pool.cc:214] Skipping call since client already timed out
0806 03:51:49.515760 (+ 130us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 03:51:49.673147 2375 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:49.516054 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2060) took 1617 ms. Trace:
I20260806 03:51:49.674283 2581 rpcz_store.cc:276] 0806 03:51:47.899137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.302921 (+403784us) service_pool.cc:227] Handling call
0806 03:51:48.303470 (+ 549us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:48.941149 (+637679us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.515756 (+574607us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b540004b854. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:49.515964 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1464570752}
W20260806 03:51:49.597399 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 144316us
I20260806 03:51:49.670023 2577 rpcz_store.cc:276] 0806 03:51:47.867741 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.569631 (+701890us) service_pool.cc:227] Handling call
0806 03:51:49.590941 (+1021310us) spinlock_profiling.cc:243] Waited 420 ms on lock 0x7b540024c7d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4aaf0 00007f0d7cacdf2a 00007f0d81a0ac6e 00007f0d7b101807 00007f0d7b101792 00007f0d7b1016c1 00007f0d7b10163c 00007f0d7b1008f2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:49.591429 (+ 488us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":965299584}
I20260806 03:51:49.596697 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1707) took 1596 ms. Trace:
I20260806 03:51:49.684640 2534 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:49.484536 2572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 972) took 2948 ms. Trace:
I20260806 03:51:49.668157 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1714) took 3843 ms. Trace:
I20260806 03:51:49.685552 2331 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:49.701601 2478 rpcz_store.cc:276] 0806 03:51:45.824961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.667215 (+3842254us) service_pool.cc:227] Handling call
0806 03:51:49.667529 (+ 314us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.667736 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:49.667765 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:49.667807 (+ 42us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:49.667917 (+ 110us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.667974 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:49.668092 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.470696 5116 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:49.598282 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.708446 2379 raft_consensus.cc:1226] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:49.555612 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2000) took 1379 ms. Trace:
W20260806 03:51:49.593896 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 355681us
I20260806 03:51:49.665443 2370 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:49.724145 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.523830 2377 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:51:49.480516 5032 raft_consensus.cc:3032] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:49.685176 2369 rpcz_store.cc:276] 0806 03:51:48.000413 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.963359 (+962946us) service_pool.cc:227] Handling call
0806 03:51:48.963663 (+ 304us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.037608 (+ 73945us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:49.052300 (+ 14692us) log.cc:842] Serialized 14 byte log entry
0806 03:51:49.594640 (+542340us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:49.594714 (+ 74us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:49.594783 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.596374 (+ 1591us) spinlock_profiling.cc:243] Waited 73.7 ms on lock 0x7b54001ea5d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:49.596443 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:49.596616 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":239920,"prepare.queue_time_us":24412,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":242335,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":94,"spinlock_wait_cycles":169402624,"thread_start_us":301968,"threads_started":1,"wal-append.queue_time_us":352977}
I20260806 03:51:49.702448 5247 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:49.768666 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.512434 5127 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:49.760138 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:49.577718 4906 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:49.704720 2572 rpcz_store.cc:276] 0806 03:51:46.535747 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:46.536147 (+ 400us) service_pool.cc:227] Handling call
0806 03:51:46.537965 (+ 1818us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.360637 (+2822672us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.484218 (+123581us) spinlock_profiling.cc:243] Waited 2.79 s on lock 0x7b540024dbd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:49.484443 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6418448768}
I20260806 03:51:49.475565 5271 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:49.778147 2232 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33786 (request call id 213) took 13309 ms (client timeout 15000 ms). Trace:
I20260806 03:51:49.481011 5245 raft_consensus.cc:3032] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:51:49.818691 3594 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 03:51:49.488543 5008 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:49.765954 2534 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:49.598881 5236 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:49.573872 5007 raft_consensus.cc:3032] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:49.594210 4993 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:49.703574 2477 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:49.572914 5018 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:49.479115 3586 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260806 03:51:49.778750 5269 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:49.769670 2480 raft_consensus.cc:1226] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
W20260806 03:51:49.478463 5262 log.cc:924] Time spent T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260806 03:51:49.515344 4797 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:49.734557 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.556913 4745 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:49.548977 3588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36484 (request call id 275) took 4066 ms (client timeout 1648 ms). Trace:
I20260806 03:51:49.781832 2581 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:49.708460 5216 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.135s
I20260806 03:51:49.776247 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:49.540043 5023 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:49.810119 2583 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:49.588721 4983 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:49.812820 2584 raft_consensus.cc:2396] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260806 03:51:49.585451 4981 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:49.703768 5281 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.148s
I20260806 03:51:49.593242 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1705) took 1593 ms. Trace:
I20260806 03:51:49.530306 5209 raft_consensus.cc:3032] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:49.761507 2479 raft_consensus.cc:1226] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 2. Current term is 3. Ops: []
W20260806 03:51:49.818208 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7970368349924684s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1569s for a response
I20260806 03:51:49.823222 2372 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260806 03:51:49.530853 5118 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:49.724517 5001 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:51:49.824591 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 259800us
I20260806 03:51:49.574615 5166 raft_consensus.cc:3032] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:51:49.830245 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 61993.7us
I20260806 03:51:49.597978 5087 consensus_queue.cc:1046] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.222s
I20260806 03:51:49.776293 5032 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:49.817751 2235 catalog_manager.cc:5495] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 2 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:49.857228 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 129) took 3756 ms. Trace:
I20260806 03:51:49.778354 5019 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:49.489020 5204 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:49.769428 5246 consensus_queue.cc:1046] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.087s
I20260806 03:51:49.776196 2482 rpcz_store.cc:276] 0806 03:51:48.176299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.259703 (+1083404us) service_pool.cc:227] Handling call
0806 03:51:49.555289 (+295586us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400250b54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:49.555523 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":320688384}
W20260806 03:51:50.055763 3542 leader_election.cc:334] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.687s (SENT)
I20260806 03:51:50.062872 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:50.044492 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 333581us
I20260806 03:51:50.079418 2377 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:50.079869 2480 rpcz_store.cc:276] 0806 03:51:46.101163 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.694783 (+3593620us) service_pool.cc:227] Handling call
0806 03:51:49.769409 (+ 74626us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.856853 (+ 87444us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.857144 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.865694 2478 raft_consensus.cc:2396] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:50.064268 3545 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:49.818179 5245 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:50.092368 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:49.814616 5163 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:49.819341 5271 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:49.843072 5007 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:50.046504 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1702) took 5743 ms. Trace:
I20260806 03:51:49.843338 5284 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:50.081617 2482 raft_consensus.cc:2362] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5. Re-sending same reply.
I20260806 03:51:50.084812 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:49.810647 5288 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:50.092252 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7799368351697922s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
I20260806 03:51:50.112933 2478 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
W20260806 03:51:50.113075 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7727368350606412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
W20260806 03:51:49.477047 2474 rpcz_store.cc:269] 0806 03:51:47.514574 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.957623 (+1443049us) service_pool.cc:227] Handling call
0806 03:51:49.447846 (+490223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.816365 4906 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:50.100596 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 131) took 3998 ms. Trace:
I20260806 03:51:50.121188 2331 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:50.121735 2479 rpcz_store.cc:276] 0806 03:51:46.102067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.694906 (+3592839us) service_pool.cc:227] Handling call
0806 03:51:49.761247 (+ 66341us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:50.100190 (+338943us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.100521 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:49.854857 5138 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:50.009902 5119 consensus_meta.cc:287] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.641s user 0.011s sys 0.000s
I20260806 03:51:50.110142 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:50.140089 2480 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:50.094719 2376 rpcz_store.cc:276] 0806 03:51:47.999739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:48.598520 (+598781us) service_pool.cc:227] Handling call
0806 03:51:48.853859 (+255339us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:48.854180 (+ 321us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:51:48.964947 (+110767us) log.cc:842] Serialized 14 byte log entry
0806 03:51:49.592750 (+627803us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:49.592831 (+ 81us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:49.592895 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.592995 (+ 100us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:49.593161 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":336516,"prepare.queue_time_us":110527,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":337953,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":109,"thread_start_us":290832,"threads_started":1,"wal-append.queue_time_us":382718}
W20260806 03:51:50.113529 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7727368350606412s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15960000000000002s for a response
I20260806 03:51:50.118714 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:50.064347 5102 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.667s user 0.013s sys 0.000s
I20260806 03:51:49.937975 5268 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:50.081570 5166 raft_consensus.cc:729] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:50.126170 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:50.126556 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.162110 2478 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:50.161835 2369 raft_consensus.cc:2362] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5. Re-sending same reply.
I20260806 03:51:50.163293 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.126204 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.123145 2481 rpcz_store.cc:276] 0806 03:51:44.302543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.350945 (+5048402us) service_pool.cc:227] Handling call
0806 03:51:49.364266 (+ 13321us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.364559 (+ 293us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:49.670266 (+305707us) log.cc:842] Serialized 14 byte log entry
0806 03:51:50.045930 (+375664us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:50.046006 (+ 76us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:50.046066 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.046143 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:50.046284 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":123784,"prepare.queue_time_us":574895,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":123949,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":159,"thread_start_us":536798,"threads_started":2}
I20260806 03:51:50.092716 5246 consensus_queue.cc:1225] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 03:51:50.081432 5001 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:50.083683 5209 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:50.142098 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7709368350915611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
W20260806 03:51:50.181200 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7259368351660669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
W20260806 03:51:50.181592 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7259368351660669s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1648s for a response
W20260806 03:51:50.181969 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7052368351724s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16710000000000003s for a response
W20260806 03:51:50.182252 2436 connection.cc:232] RPC call timeout handler was delayed by -2.6800368351396173s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16990000000000002s for a response
W20260806 03:51:50.182610 2436 connection.cc:232] RPC call timeout handler was delayed by -2.6476368349976838s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17350000000000002s for a response
W20260806 03:51:50.182876 2436 connection.cc:232] RPC call timeout handler was delayed by -2.6215368350967765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
W20260806 03:51:50.183115 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5909368351567537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
W20260806 03:51:50.183369 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5909368351567537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
W20260806 03:51:50.183611 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5378368350211531s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
I20260806 03:51:50.141255 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:50.183841 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5270368349738419s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1869s for a response
W20260806 03:51:50.185873 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5045368350110948s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20260806 03:51:50.186149 2436 connection.cc:232] RPC call timeout handler was delayed by -2.466736834961921s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1936s for a response
W20260806 03:51:50.186407 2436 connection.cc:232] RPC call timeout handler was delayed by -2.4541368351783603s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.195s for a response
W20260806 03:51:50.186646 2436 connection.cc:232] RPC call timeout handler was delayed by -2.4298368350137025s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19770000000000001s for a response
W20260806 03:51:50.186892 2436 connection.cc:232] RPC call timeout handler was delayed by -2.4091368350200355s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2s for a response
W20260806 03:51:50.187125 2436 connection.cc:232] RPC call timeout handler was delayed by -2.3542368351481855s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2061s for a response
W20260806 03:51:50.187367 2436 connection.cc:232] RPC call timeout handler was delayed by -2.3245368350762874s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
W20260806 03:51:50.187640 2436 connection.cc:232] RPC call timeout handler was delayed by -2.2966368349734694s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21250000000000002s for a response
W20260806 03:51:50.051316 5218 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.004s sys 0.007s
W20260806 03:51:50.187887 2436 connection.cc:232] RPC call timeout handler was delayed by -2.2165368350688368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
W20260806 03:51:50.191694 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1895368350669742s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22440000000000004s for a response
I20260806 03:51:50.159509 2372 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dad99dac270543b19675707d75149e78"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
W20260806 03:51:50.192744 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1310368350241333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23090000000000002s for a response
I20260806 03:51:50.119771 5284 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:50.092329 5127 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260806 03:51:50.173180 2482 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:50.193434 2372 raft_consensus.cc:595] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
W20260806 03:51:49.816818 2232 rpcz_store.cc:269] 0806 03:51:36.450234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:36.450409 (+ 175us) service_pool.cc:227] Handling call
0806 03:51:49.759706 (+13309297us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:42.903729 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:43.056991 (+153262us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:43.064947 (+ 7956us) write_op.cc:435] Acquired schema lock
0806 03:51:43.077519 (+ 12572us) tablet.cc:660] Decoding operations
0806 03:51:43.418080 (+340561us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:43.418231 (+ 151us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:43.418261 (+ 30us) tablet.cc:683] Acquiring locks for 7 operations
0806 03:51:43.418967 (+ 706us) tablet.cc:699] Row locks acquired
0806 03:51:43.419018 (+ 51us) write_op.cc:260] PREPARE: finished
0806 03:51:43.419215 (+ 197us) write_op.cc:270] Start()
0806 03:51:43.419377 (+ 162us) write_op.cc:276] Timestamp: P: 1785988303419140 usec, L: 0
0806 03:51:43.419418 (+ 41us) op_driver.cc:331] REPLICATION: starting
0806 03:51:43.526908 (+107490us) log.cc:842] Serialized 2869 byte log entry
0806 03:51:44.739465 (+1212557us) op_driver.cc:444] REPLICATION: finished
0806 03:51:45.270660 (+531195us) write_op.cc:301] APPLY: starting
0806 03:51:45.282082 (+ 11422us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:45.632456 (+350374us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:45.766302 (+133846us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:46.235712 (+469410us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:46.236176 (+ 464us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0806 03:51:46.236248 (+ 72us) write_op.cc:312] APPLY: finished
0806 03:51:46.236764 (+ 516us) log.cc:842] Serialized 59 byte log entry
0806 03:51:46.366250 (+129486us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:46.366459 (+ 209us) write_op.cc:454] Released schema lock
0806 03:51:46.366555 (+ 96us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:46.411168 (+ 44613us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:46.416155 (+ 4987us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":387568,"apply.run_cpu_time_us":43154,"apply.run_wall_time_us":1225493,"num_ops":7,"prepare.queue_time_us":828880,"prepare.run_cpu_time_us":41846,"prepare.run_wall_time_us":816612,"raft.queue_time_us":540823,"raft.run_cpu_time_us":8491,"raft.run_wall_time_us":228955,"replication_time_us":1308202,"thread_start_us":1398745,"threads_started":5,"wal-append.queue_time_us":508985,"wal-append.run_cpu_time_us":22293,"wal-append.run_wall_time_us":2729446}]]}
I20260806 03:51:50.174674 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.197420 2478 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 1.
I20260806 03:51:50.198625 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.199424 2478 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:50.177219 2368 raft_consensus.cc:1261] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260806 03:51:50.134505 4978 consensus_meta.cc:287] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.730s user 0.010s sys 0.004s
I20260806 03:51:49.848111 4856 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.277s
I20260806 03:51:50.158077 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:50.205598 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:50.161474 5141 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.756s user 0.009s sys 0.004s
I20260806 03:51:50.195523 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.213812 2479 raft_consensus.cc:2362] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4. Re-sending same reply.
I20260806 03:51:50.167171 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:50.084437 5019 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.134905 5033 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:50.204093 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:50.196425 5218 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.718s user 0.004s sys 0.010s
I20260806 03:51:50.215299 4856 consensus_queue.cc:1225] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer cf6042e85bef47c4aac9d064d91501d1 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 03:51:50.216600 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:50.219816 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.161808 5102 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:50.237252 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:50.223567 2375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.004s sys 0.017s
I20260806 03:51:50.140383 5313 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:50.238831 2375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.528s user 0.004s sys 0.017s
I20260806 03:51:50.239307 5313 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:50.239964 5313 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.241022 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:50.227314 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 125227us
I20260806 03:51:50.180295 5001 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:49.848716 5236 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.216315 4981 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:50.100095 5163 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.128530 5288 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.232787 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:50.231287 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 149517us
I20260806 03:51:50.312395 2480 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:50.234313 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 72148.5us
I20260806 03:51:50.235212 2478 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:50.207473 5284 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:50.195365 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:50.193189 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1040368350222707s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2339s for a response
W20260806 03:51:50.098923 3588 rpcz_store.cc:269] 0806 03:51:45.482991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:45.496140 (+ 13149us) service_pool.cc:227] Handling call
0806 03:51:49.548708 (+4052568us) spinlock_profiling.cc:243] Waited 3.31 s on lock 0x7b540027a954. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:49.548864 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7613226240}
I20260806 03:51:49.923588 5269 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.270990 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:50.273016 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:50.338245 2369 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:50.338515 2368 raft_consensus.cc:2396] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 3 because replica has last-logged OpId of term: 1 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 2.
I20260806 03:51:50.336225 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:50.216775 5312 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:50.279742 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:50.343220 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:50.343852 2373 raft_consensus.cc:2362] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3. Re-sending same reply.
I20260806 03:51:50.344735 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:50.345082 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:50.280256 3542 leader_election.cc:334] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.851s (SENT)
I20260806 03:51:50.347199 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:50.287878 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:50.236531 5033 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:50.291385 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 168614us
I20260806 03:51:50.165649 5268 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:50.292984 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 127696us
W20260806 03:51:50.353355 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 73169.9us
I20260806 03:51:50.293713 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:50.182152 5246 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:50.303409 2377 raft_consensus.cc:1226] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:50.175757 5138 consensus_queue.cc:1046] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260806 03:51:50.216368 4978 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.314642 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.317044 2534 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:50.315510 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.321560 2429 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:51:50.324296 2332 leader_election.cc:334] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.673s (SENT)
W20260806 03:51:50.309181 5292 log.cc:924] Time spent T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.342s user 0.000s sys 0.002s
I20260806 03:51:50.364835 5278 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:50.366160 5268 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:50.367178 2480 raft_consensus.cc:1226] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:50.348279 5312 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:50.368388 5312 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:50.349527 5319 consensus_queue.cc:1046] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260806 03:51:50.370118 5318 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 03:51:50.325549 2436 connection.cc:232] RPC call timeout handler was delayed by -1.6432368350215256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2851s for a response
W20260806 03:51:50.377382 2436 connection.cc:232] RPC call timeout handler was delayed by -1.6342368351761252s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2861s for a response
W20260806 03:51:50.377839 2436 connection.cc:232] RPC call timeout handler was delayed by -1.6045368351042271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28940000000000005s for a response
W20260806 03:51:50.316802 5289 log.cc:924] Time spent T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.449s user 0.002s sys 0.000s
I20260806 03:51:50.339941 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.254169 5218 consensus_queue.cc:1046] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.034s
I20260806 03:51:50.228921 5245 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.406911 4993 raft_consensus.cc:3032] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:51:50.407469 4993 raft_consensus.cc:729] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:50.283249 5166 consensus_queue.cc:260] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.408571 5166 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:50.408464 4993 consensus_queue.cc:260] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.410020 4993 raft_consensus.cc:3037] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:50.222386 5007 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.412740 5130 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:50.352078 2368 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:50.361442 2373 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260806 03:51:50.361779 2376 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 03:51:50.415912 5302 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.232s user 0.000s sys 0.001s
I20260806 03:51:50.363198 5304 consensus_queue.cc:1035] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:50.367924 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1963) took 4254 ms. Trace:
I20260806 03:51:50.417838 2480 rpcz_store.cc:276] 0806 03:51:46.113212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:50.362252 (+4249040us) service_pool.cc:227] Handling call
0806 03:51:50.366987 (+ 4735us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:50.367631 (+ 644us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.367825 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:50.377276 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 60322.9us
I20260806 03:51:50.378935 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:50.355501 2478 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:50.296566 4981 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:50.340664 2372 raft_consensus.cc:2396] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:50.376204 5246 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.385790 2479 raft_consensus.cc:2396] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:50.348767 2474 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:50.413220 5130 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:50.359282 2369 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:51:50.420866 5067 raft_consensus.cc:3032] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:50.421001 2481 raft_consensus.cc:2396] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260806 03:51:50.394832 2436 connection.cc:232] RPC call timeout handler was delayed by -1.4992368351668119s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30110000000000003s for a response
I20260806 03:51:50.457880 2436 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:50.456980 5067 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:50.462236 5067 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.439805 4981 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:50.458302 5130 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:51:50.469265 5307 log.cc:924] Time spent T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.158s user 0.001s sys 0.000s
W20260806 03:51:50.443091 5317 consensus_queue.cc:1159] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fe3d17f4ad984b9088a268d58d87469c" responder_term: 4 status { last_received { term: 3 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:51:50.824388 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1726) took 1453 ms. Trace:
W20260806 03:51:50.828105 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 370340us
I20260806 03:51:50.830017 2436 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:50.832880 2478 raft_consensus.cc:1226] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:50.834022 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1971) took 4582 ms. Trace:
I20260806 03:51:50.834527 2478 rpcz_store.cc:276] 0806 03:51:46.251936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:50.832158 (+4580222us) service_pool.cc:227] Handling call
0806 03:51:50.832632 (+ 474us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:50.833691 (+ 1059us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.833951 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:50.837949 2429 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:50.837097 2478 raft_consensus.cc:1226] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:51:50.839326 2538 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:50.842840 2538 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:50.441812 5313 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:50.502215 5308 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.212s user 0.000s sys 0.001s
W20260806 03:51:50.846993 2484 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.884s user 0.008s sys 0.011s
W20260806 03:51:50.848559 2484 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.886s user 0.008s sys 0.012s
W20260806 03:51:50.642311 4983 consensus_meta.cc:287] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.533s user 0.011s sys 0.003s
W20260806 03:51:50.537052 5018 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.659s user 0.009s sys 0.005s
I20260806 03:51:50.445612 5218 raft_consensus.cc:3032] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:51:50.505329 5316 log.cc:924] Time spent T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20260806 03:51:50.507988 5269 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:50.574111 5087 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:50.504134 5001 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.566205 5209 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.822353 5246 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:50.470117 5236 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:50.840740 5007 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:50.511114 5019 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:50.849388 5303 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:50.627099 5278 raft_consensus.cc:686] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:50.841836 5297 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:50.843367 5067 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:50.506494 5163 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:50.847537 5095 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:50.837517 5304 raft_consensus.cc:2726] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:50.842594 5033 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:50.884507 5315 log.cc:924] Time spent T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.592s user 0.006s sys 0.007s
I20260806 03:51:50.470361 5321 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:50.885129 5247 raft_consensus.cc:3032] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:50.915036 5247 raft_consensus.cc:729] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:50.915099 5321 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.916186 5247 consensus_queue.cc:260] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.917598 5247 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:50.919263 5312 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.550s user 0.004s sys 0.006s
I20260806 03:51:50.844751 5102 consensus_queue.cc:1035] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260806 03:51:50.912387 5303 raft_consensus.cc:686] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:50.922643 5322 raft_consensus.cc:979] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:50.923375 5334 raft_consensus.cc:1067] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Signalling peer dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:51:50.923902 5318 raft_consensus.cc:3032] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:50.924471 5318 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:50.925704 5318 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:50.926920 5166 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.518s user 0.003s sys 0.006s
I20260806 03:51:50.904953 5277 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:50.873168 5245 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:50.890281 5218 raft_consensus.cc:729] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:50.917675 5321 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:50.907819 5288 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:50.927659 5277 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.938072 5277 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:50.927654 5318 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:50.939121 5284 consensus_meta.cc:287] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.602s user 0.005s sys 0.007s
I20260806 03:51:50.923322 5303 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.553440 5271 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:50.943404 5271 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:50.468068 5320 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.451s
W20260806 03:51:50.950816 5305 log.cc:924] Time spent T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.511s user 0.005s sys 0.010s
I20260806 03:51:50.919708 5312 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.947489 5284 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.130802 5284 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:51.132376 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 289417us
I20260806 03:51:50.825027 2370 rpcz_store.cc:276] 0806 03:51:49.371384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.758243 (+386859us) service_pool.cc:227] Handling call
0806 03:51:49.758808 (+ 565us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:49.761605 (+ 2797us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:49.761641 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:49.817132 (+ 55491us) log.cc:842] Serialized 26 byte log entry
0806 03:51:50.217548 (+400416us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:50.217592 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.217662 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:50.824085 (+606423us) raft_consensus.cc:1671] finished
0806 03:51:50.824118 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:50.824300 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2251,"raft.queue_time_us":19230,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146,"thread_start_us":352594,"threads_started":1,"wal-append.queue_time_us":553335}
I20260806 03:51:51.007331 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1724) took 1705 ms. Trace:
I20260806 03:51:51.140890 2375 rpcz_store.cc:276] 0806 03:51:49.302229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.673780 (+371551us) service_pool.cc:227] Handling call
0806 03:51:49.709631 (+ 35851us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:49.710023 (+ 392us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:49.710052 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:49.710596 (+ 544us) log.cc:842] Serialized 26 byte log entry
0806 03:51:50.239365 (+528769us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:50.239430 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.239511 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:51.006658 (+767147us) raft_consensus.cc:1671] finished
0806 03:51:51.006718 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.007032 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1055,"raft.run_cpu_time_us":221,"raft.run_wall_time_us":53194,"thread_start_us":528388,"threads_started":1,"wal-append.queue_time_us":625838}
I20260806 03:51:51.142514 2538 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:51.007133 2235 catalog_manager.cc:5495] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 3 to 4, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 4 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:51:50.502889 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2008) took 1433 ms. Trace:
I20260806 03:51:51.144395 2477 rpcz_store.cc:276] 0806 03:51:49.069232 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.296987 (+227755us) service_pool.cc:227] Handling call
0806 03:51:50.502621 (+1205634us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b54003511d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:50.502797 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":378303360}
I20260806 03:51:51.145514 5248 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:50.995965 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.016510 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1725) took 1713 ms. Trace:
I20260806 03:51:51.017721 2480 raft_consensus.cc:1226] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: []
I20260806 03:51:51.019835 2483 raft_consensus.cc:2396] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:50.849390 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1713) took 5084 ms. Trace:
I20260806 03:51:50.840543 2436 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.153264 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:51.153848 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 313244us
I20260806 03:51:51.154444 2376 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:51.024894 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1711) took 6231 ms. Trace:
I20260806 03:51:51.026466 2482 raft_consensus.cc:2396] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:51.045545 2575 raft_consensus.cc:2396] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:51.050271 2429 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:51.054649 2481 raft_consensus.cc:1226] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:51.055045 3594 raft_consensus.cc:1226] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:50.508641 4978 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:51.056414 5007 consensus_queue.cc:1046] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:51.057101 5035 raft_consensus.cc:3032] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:51:51.164954 5035 raft_consensus.cc:729] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:51.165304 5007 raft_consensus.cc:3032] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:51:51.165953 5007 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:51.061020 4856 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:50.933518 5116 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.165s user 0.015s sys 0.000s
I20260806 03:51:51.173084 2583 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:51.167448 5007 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.172960 5116 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.174551 5007 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.068140 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:51:50.910960 4993 consensus_meta.cc:287] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.500s user 0.011s sys 0.004s
I20260806 03:51:51.176409 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.072902 4745 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.177623 4745 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.167038 2484 rpcz_store.cc:276] 0806 03:51:45.765318 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.666755 (+3901437us) service_pool.cc:227] Handling call
0806 03:51:49.667231 (+ 476us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:49.667465 (+ 234us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:51:49.950253 (+282788us) log.cc:842] Serialized 14 byte log entry
0806 03:51:50.848986 (+898733us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:50.849045 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:50.849097 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.849180 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:50.849326 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":333826,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":66,"raft.queue_time_us":38330,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":4331,"thread_start_us":1127784,"threads_started":2,"wal-append.queue_time_us":998735}
I20260806 03:51:50.553913 5032 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:51.181355 2483 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2002) took 2963 ms (client timeout 1587 ms). Trace:
W20260806 03:51:51.182008 2483 rpcz_store.cc:269] 0806 03:51:48.218024 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.144710 (+926686us) service_pool.cc:227] Handling call
0806 03:51:51.180969 (+2036259us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54001ab154. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:51.181207 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4306411392}
I20260806 03:51:51.075145 2369 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260806 03:51:51.183314 5130 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:51.077280 2233 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33778 (request call id 215) took 12926 ms (client timeout 15000 ms). Trace:
W20260806 03:51:51.184628 2233 rpcz_store.cc:269] 0806 03:51:38.150319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:38.150489 (+ 170us) service_pool.cc:227] Handling call
0806 03:51:39.313600 (+1163111us) spinlock_profiling.cc:243] Waited 942 ms on lock 0x7b54000e3154. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4aaf0 00007f0d7cacdf2a 00007f0d82a1fd6c 00007f0d82b19c3e 00007f0d80e2bfb4 00007f0d80e2bf42 00007f0d80e2be71 00007f0d80e2bdec 00007f0d80e2b0a2 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:51.076869 (+11763269us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":181925,"reported_tablets":5,"spinlock_wait_cycles":2166629632}
I20260806 03:51:51.075006 5113 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.080423 5119 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:50.891618 5087 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:50.904744 4986 raft_consensus.cc:2781] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:51.088066 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.089332 2377 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260806 03:51:51.166731 5035 consensus_queue.cc:260] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.090548 5141 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.090863 5247 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.103605 2479 raft_consensus.cc:1226] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:51.104156 5321 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:51.106192 5336 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.106328 5338 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.075084 5001 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:51.106664 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.116724 5218 consensus_queue.cc:260] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.120736 5209 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:51.124189 2474 raft_consensus.cc:1226] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 03:51:51.183197 2373 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:40728
I20260806 03:51:51.220657 2373 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:51:51.199316 4981 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.221081 2373 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:51.221464 5329 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:51.222285 5329 raft_consensus.cc:686] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:51:51.204123 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 61816.3us
I20260806 03:51:51.125845 5268 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260806 03:51:51.171483 4856 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.224180 2436 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:51.124873 2374 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.007s sys 0.010s
I20260806 03:51:51.223721 5329 consensus_queue.cc:237] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:51.226176 2374 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.804s user 0.008s sys 0.011s
I20260806 03:51:51.226336 4856 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:51.188674 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1976) took 4846 ms. Trace:
I20260806 03:51:51.227283 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1729) took 1741 ms. Trace:
I20260806 03:51:51.134724 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:50.847072 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1975) took 4539 ms. Trace:
I20260806 03:51:51.146134 2477 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "fe3d17f4ad984b9088a268d58d87469c"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:51:51.147167 2379 rpcz_store.cc:276] 0806 03:51:49.303375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.709261 (+405886us) service_pool.cc:227] Handling call
0806 03:51:49.780329 (+ 71068us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:49.780761 (+ 432us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:49.780798 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:49.816891 (+ 36093us) log.cc:842] Serialized 26 byte log entry
0806 03:51:50.200374 (+383483us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:50.200434 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.200542 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:50.883805 (+683263us) spinlock_profiling.cc:243] Waited 8.73 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:51.015702 (+131897us) raft_consensus.cc:1671] finished
0806 03:51:51.015752 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.016230 (+ 478us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":36673,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":8273,"spinlock_wait_cycles":20073728,"thread_start_us":261985,"threads_started":1,"wal-append.queue_time_us":436737}
W20260806 03:51:51.161995 2575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 986) took 1993 ms (client timeout 1878 ms). Trace:
I20260806 03:51:51.162637 5330 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:51.155964 2476 rpcz_store.cc:276] 0806 03:51:44.793320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.444170 (+4650850us) service_pool.cc:227] Handling call
0806 03:51:49.456221 (+ 12051us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:49.517807 (+ 61586us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:49.517841 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:49.536438 (+ 18597us) log.cc:842] Serialized 26 byte log entry
0806 03:51:49.848399 (+311961us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:49.851713 (+ 3314us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:49.851816 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:50.950027 (+1098211us) spinlock_profiling.cc:243] Waited 7.79 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:51.023954 (+ 73927us) raft_consensus.cc:1671] finished
0806 03:51:51.023999 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.024581 (+ 582us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":113805,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":121,"spinlock_wait_cycles":17922432,"thread_start_us":363583,"threads_started":2,"wal-append.queue_time_us":796492}
I20260806 03:51:51.212141 5141 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.212930 5119 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:51.167208 4906 consensus_meta.cc:287] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.010s user 0.007s sys 0.012s
I20260806 03:51:51.204424 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 134) took 4762 ms. Trace:
I20260806 03:51:51.173420 3594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 343) took 1234 ms. Trace:
I20260806 03:51:51.238559 2484 rpcz_store.cc:276] 0806 03:51:46.442038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.179267 (+4737229us) service_pool.cc:227] Handling call
0806 03:51:51.179766 (+ 499us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.193856 (+ 14090us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:51.193901 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:51.203764 (+ 9863us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:51.203897 (+ 133us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.204044 (+ 147us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.204288 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:51.077723 5118 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.988s user 0.017s sys 0.000s
I20260806 03:51:51.183353 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:51.125330 5320 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:51.184208 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1970) took 4932 ms. Trace:
I20260806 03:51:51.144927 5278 consensus_queue.cc:237] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:51.103235 5304 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260806 03:51:51.226999 2481 rpcz_store.cc:276] 0806 03:51:46.323936 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.053105 (+4729169us) service_pool.cc:227] Handling call
0806 03:51:51.053992 (+ 887us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.158719 (+104727us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.168806 (+ 10087us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.137868 5312 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:51.229233 2575 rpcz_store.cc:269] 0806 03:51:49.168310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.168457 (+ 147us) service_pool.cc:227] Handling call
0806 03:51:51.161664 (+1993207us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b540028f0d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:51.161887 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4263441664}
I20260806 03:51:51.214596 5338 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.239297 3594 rpcz_store.cc:276] 0806 03:51:49.938875 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.939150 (+ 275us) service_pool.cc:227] Handling call
0806 03:51:50.555384 (+616234us) spinlock_profiling.cc:243] Waited 2.37 ms on lock 0x7b5800000f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:50.555767 (+ 383us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.171384 (+615617us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.172996 (+ 1612us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54007701d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:51.173326 (+ 330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1151752960}
I20260806 03:51:51.256664 2480 rpcz_store.cc:276] 0806 03:51:46.251610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:50.832134 (+4580524us) service_pool.cc:227] Handling call
0806 03:51:51.017048 (+184914us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.149978 (+132930us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.184114 (+ 34136us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.212646 5247 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.214063 5336 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:51.218758 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 168878us
I20260806 03:51:51.216043 5218 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:51.212386 5035 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:51.215268 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1745) took 1218 ms. Trace:
I20260806 03:51:51.212929 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1972) took 4960 ms. Trace:
I20260806 03:51:51.209143 2372 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:51.247861 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.187880 5116 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:51.227804 2374 rpcz_store.cc:276] 0806 03:51:49.486244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.760610 (+274366us) service_pool.cc:227] Handling call
0806 03:51:49.824811 (+ 64201us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:49.825144 (+ 333us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:49.825169 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:49.846833 (+ 21664us) log.cc:842] Serialized 26 byte log entry
0806 03:51:50.422015 (+575182us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b180038e7e0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b7f345 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9ce22 00007f0d7c7465bb 00007f0d7c74c063 00007f0d7c74c411 00007f0d7ca995cf
0806 03:51:51.226769 (+804754us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:51.226831 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.226925 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:51.226965 (+ 40us) raft_consensus.cc:1671] finished
0806 03:51:51.226973 (+ 8us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.227222 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":23353,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":20928,"spinlock_wait_cycles":6562048,"thread_start_us":804440,"threads_started":1,"wal-append.queue_time_us":1348649}
I20260806 03:51:51.280268 2369 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260806 03:51:51.290920 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 295763us
I20260806 03:51:51.210597 5087 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.296144 5317 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.103s
I20260806 03:51:51.285977 2478 rpcz_store.cc:276] 0806 03:51:46.299480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:50.835588 (+4536108us) service_pool.cc:227] Handling call
0806 03:51:50.835960 (+ 372us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:50.838877 (+ 2917us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:50.839075 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.297760 5338 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:51.260325 5268 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:51.299278 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1726) took 4646 ms. Trace:
I20260806 03:51:51.299408 5268 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.285935 5320 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.302822 4981 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.300400 5268 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:51.282487 3542 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.902s (SENT) [suppressed 41 similar messages]
I20260806 03:51:51.284004 2479 rpcz_store.cc:276] 0806 03:51:46.252280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.035775 (+4783495us) service_pool.cc:227] Handling call
0806 03:51:51.091494 (+ 55719us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.212531 (+121037us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.212804 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.283286 2482 rpcz_store.cc:276] 0806 03:51:49.997037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:50.195221 (+198184us) service_pool.cc:227] Handling call
0806 03:51:51.214951 (+1019730us) spinlock_profiling.cc:243] Waited 808 ms on lock 0x7b54002b2354. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:51.215172 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1858848512}
I20260806 03:51:51.299926 2478 rpcz_store.cc:276] 0806 03:51:46.652717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.297895 (+4645178us) service_pool.cc:227] Handling call
0806 03:51:51.298373 (+ 478us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.298711 (+ 338us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:51.298747 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:51.298826 (+ 79us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:51.298890 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.299005 (+ 115us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.299162 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.308701 2375 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.257571 4745 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:51.258438 2477 raft_consensus.cc:595] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Received request to transfer leadership to TS fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.218415 5032 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:51.306942 2476 raft_consensus.cc:1226] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:51.315734 5247 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:51.319581 3593 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:51.340356 5119 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.340581 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1720) took 4955 ms. Trace:
I20260806 03:51:51.340986 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1728) took 4682 ms. Trace:
I20260806 03:51:51.341861 3593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 337) took 2178 ms. Trace:
I20260806 03:51:51.320173 4986 raft_consensus.cc:686] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:51.321709 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 44629.9us
I20260806 03:51:51.323348 5001 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.278169 4906 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.326009 5336 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.348469 5067 consensus_queue.cc:1046] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:51.349515 5067 raft_consensus.cc:3032] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:51.349078 4986 consensus_queue.cc:237] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.349943 5067 raft_consensus.cc:729] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:51.351153 4906 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.352079 3588 raft_consensus.cc:2379] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:51.351420 5067 consensus_queue.cc:260] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.325978 5268 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.328384 5095 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.331102 3547 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.303758 5320 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:51.332999 2480 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:51.333457 2477 raft_consensus.cc:603] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:51:51.361382 4993 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:51.358554 5095 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.340094 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1977) took 4828 ms. Trace:
I20260806 03:51:51.362212 5130 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:51.355307 5087 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.362485 2476 rpcz_store.cc:276] 0806 03:51:46.511661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.235665 (+4724004us) service_pool.cc:227] Handling call
0806 03:51:51.237901 (+ 2236us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.339712 (+101811us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.339994 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.363257 2477 raft_consensus.cc:1226] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:51.363489 5087 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.364624 2476 raft_consensus.cc:1261] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260806 03:51:51.365259 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1733) took 1663 ms. Trace:
I20260806 03:51:51.365947 5087 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:51.366178 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 58678.3us
W20260806 03:51:51.366695 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 77822.6us
I20260806 03:51:51.371762 5001 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:51.372354 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 216669us
W20260806 03:51:51.319686 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 131242us
I20260806 03:51:51.373476 2436 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.374408 5001 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:51.376241 5304 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.341449 2482 rpcz_store.cc:276] 0806 03:51:46.659037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.338313 (+4679276us) service_pool.cc:227] Handling call
0806 03:51:51.340324 (+ 2011us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.340594 (+ 270us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:51.340624 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:51.340699 (+ 75us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:51.340731 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.340789 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.340898 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.377058 5304 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.377604 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:51.362207 4993 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.379163 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:51.379617 5304 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.348280 5097 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.213377 5321 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.352317 5236 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:51.343500 5141 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.359910 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 135) took 4737 ms. Trace:
I20260806 03:51:51.372453 5247 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:51.376333 5067 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:51.342298 3593 rpcz_store.cc:276] 0806 03:51:49.163509 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:49.165114 (+ 1605us) service_pool.cc:227] Handling call
0806 03:51:49.227865 (+ 62751us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.340804 (+2112939us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.341595 (+ 791us) spinlock_profiling.cc:243] Waited 2.05 s on lock 0x7b540067e8d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:51.341764 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4710505856}
I20260806 03:51:51.379889 5095 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:51.341179 2474 rpcz_store.cc:276] 0806 03:51:46.385034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.082568 (+4697534us) service_pool.cc:227] Handling call
0806 03:51:51.083418 (+ 850us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:51.220312 (+136894us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.340209 (+119897us) spinlock_profiling.cc:243] Waited 77.6 ms on lock 0x7b5400432720. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7ca55730 00007f0d7cac8b42 00007f0d7cac4f26 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:51:51.340493 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178433280}
I20260806 03:51:51.406551 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:51.407516 2474 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:51.366726 5087 consensus_queue.cc:1046] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260806 03:51:51.409089 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:51.410046 2474 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:51.365595 2476 rpcz_store.cc:276] 0806 03:51:49.702268 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.363696 (+1661428us) service_pool.cc:227] Handling call
0806 03:51:51.364147 (+ 451us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.365018 (+ 871us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.365206 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.343099 2375 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:51.379845 2482 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.384158 4993 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:51.414309 3547 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:51.384745 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1727) took 4729 ms. Trace:
I20260806 03:51:51.386286 5329 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:51.387565 2480 rpcz_store.cc:276] 0806 03:51:46.623035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.280375 (+4657340us) service_pool.cc:227] Handling call
0806 03:51:51.332662 (+ 52287us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.359606 (+ 26944us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.359836 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":36608}
I20260806 03:51:51.398543 2436 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:51.400899 5097 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.401427 5247 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.382997 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1732) took 1662 ms. Trace:
I20260806 03:51:51.416177 2479 rpcz_store.cc:276] 0806 03:51:46.655462 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.339090 (+4683628us) service_pool.cc:227] Handling call
0806 03:51:51.381345 (+ 42255us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.384276 (+ 2931us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:51.384312 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:51.384354 (+ 42us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:51.384446 (+ 92us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.384520 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.384654 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.422464 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260806 03:51:51.423933 2476 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:51:51.388088 5268 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.394388 5007 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:51.451736 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 143) took 1528 ms. Trace:
I20260806 03:51:51.460134 2476 rpcz_store.cc:276] 0806 03:51:49.923478 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.422868 (+1499390us) service_pool.cc:227] Handling call
0806 03:51:51.423429 (+ 561us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.451345 (+ 27916us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.451661 (+ 316us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.411547 5119 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:51.467897 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:51.469647 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1723) took 4992 ms. Trace:
I20260806 03:51:51.470116 2483 rpcz_store.cc:276] 0806 03:51:46.476823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.183092 (+4706269us) service_pool.cc:227] Handling call
0806 03:51:51.183572 (+ 480us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.238287 (+ 54715us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:51:51.384650 (+146363us) log.cc:842] Serialized 14 byte log entry
0806 03:51:51.468508 (+ 83858us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:51.468827 (+ 319us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:51.468955 (+ 128us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.469112 (+ 157us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.469522 (+ 410us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":154664,"threads_started":2}
I20260806 03:51:51.473232 2538 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:51.478116 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:51.480762 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:51.481537 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:51.480655 2479 raft_consensus.cc:1226] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:51.486032 2584 raft_consensus.cc:2396] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:51.458038 5236 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.436302 2477 rpcz_store.cc:276] 0806 03:51:49.701939 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.362264 (+1660325us) service_pool.cc:227] Handling call
0806 03:51:51.362774 (+ 510us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.363797 (+ 1023us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.363991 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.485437 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:51.493607 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 117682us
I20260806 03:51:51.496583 2575 raft_consensus.cc:1261] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:51.496894 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:51.497716 2370 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:51.500754 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:51.501696 2572 raft_consensus.cc:2362] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3. Re-sending same reply.
I20260806 03:51:51.501821 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:51:51.505838 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.491360 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.504307 3594 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:51.510843 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.521752 2377 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:51.520148 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 91621.1us
I20260806 03:51:51.527729 2369 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:51.502130 5321 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.530373 5247 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:51.536528 2332 leader_election.cc:334] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.032s (SENT)
W20260806 03:51:51.536834 5245 consensus_meta.cc:287] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.560s user 0.013s sys 0.003s
I20260806 03:51:51.537343 2436 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.537570 4993 raft_consensus.cc:3032] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Stepping down as leader of term 4
W20260806 03:51:51.540267 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 126102us
I20260806 03:51:51.540593 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.557089 2572 raft_consensus.cc:2396] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:51.542881 5330 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:51.544905 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 81482us
I20260806 03:51:51.545567 5209 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:51.540412 5119 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:51.564776 2374 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:51.551268 2583 raft_consensus.cc:2362] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3. Re-sending same reply.
I20260806 03:51:51.485296 5141 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:51.477035 5268 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.555573 2332 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.552933 2373 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.558532 2480 raft_consensus.cc:1261] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260806 03:51:51.547816 2581 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:51.558524 4993 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:51.569064 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1748) took 1504 ms. Trace:
I20260806 03:51:51.569300 2481 raft_consensus.cc:1226] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:51:51.569466 2480 rpcz_store.cc:276] 0806 03:51:50.064895 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.464084 (+1399189us) service_pool.cc:227] Handling call
0806 03:51:51.509274 (+ 45190us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.568771 (+ 59497us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.569005 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:51.570143 4993 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.571738 2482 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:51.564085 5087 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.573904 2373 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:51.575599 5360 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.065s user 0.002s sys 0.000s
I20260806 03:51:51.564491 5119 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:51.578881 5119 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.574087 5087 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.580222 5141 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.563930 5236 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.582926 5141 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.583230 2375 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.583945 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:51.584470 5236 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.588455 4906 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.588727 2534 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:51.589164 4906 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.590524 5001 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:51.563244 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.486002 5329 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.600068 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.597348 2375 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:51.606582 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 327154us
I20260806 03:51:51.575613 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 137) took 2248 ms. Trace:
I20260806 03:51:51.572182 4993 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.590955 5001 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:51.561682 5209 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:51.622465 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1982) took 5041 ms. Trace:
I20260806 03:51:51.565143 5067 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:51:51.623214 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 98943.4us
I20260806 03:51:51.634348 2538 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:51.633656 2484 rpcz_store.cc:276] 0806 03:51:46.581094 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.275113 (+4694019us) service_pool.cc:227] Handling call
0806 03:51:51.304627 (+ 29514us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.305101 (+ 474us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:51:51.475429 (+170328us) log.cc:842] Serialized 14 byte log entry
0806 03:51:51.621558 (+146129us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:51.621689 (+ 131us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:51.621783 (+ 94us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.621977 (+ 194us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.622319 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":101,"thread_start_us":283639,"threads_started":2}
I20260806 03:51:51.613803 2478 rpcz_store.cc:276] 0806 03:51:49.327187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.332426 (+2005239us) service_pool.cc:227] Handling call
0806 03:51:51.332918 (+ 492us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:51.333287 (+ 369us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:51:51.456357 (+123070us) log.cc:842] Serialized 14 byte log entry
0806 03:51:51.575211 (+118854us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:51.575279 (+ 68us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:51:51.575335 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.575424 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:51.575548 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8976,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":134,"thread_start_us":195306,"threads_started":2}
I20260806 03:51:51.649201 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:51.648334 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 49409.6us
I20260806 03:51:51.617661 5087 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:51.620988 5001 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.621408 5236 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:51.636794 5304 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:51:51.665352 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 269519us
I20260806 03:51:51.635946 5209 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:51.672142 5248 consensus_meta.cc:287] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.526s user 0.013s sys 0.003s
I20260806 03:51:51.667642 2584 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:51.677186 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:51.677505 2583 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:51.688450 5281 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.688884 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:51.689195 5281 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.689914 3588 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:51.693115 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54212 (request call id 447) took 1515 ms. Trace:
I20260806 03:51:51.693521 2476 rpcz_store.cc:276] 0806 03:51:50.177598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.476741 (+1299143us) service_pool.cc:227] Handling call
0806 03:51:51.692336 (+215595us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54000af0d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4aaf0 00007f0d7cacdf2a 00007f0d81a0ac6e 00007f0d7b101807 00007f0d7b101792 00007f0d7b1016c1 00007f0d7b10163c 00007f0d7b1008f2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:51.693008 (+ 672us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":328947072}
W20260806 03:51:51.695534 5358 log.cc:924] Time spent T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.210s user 0.001s sys 0.000s
I20260806 03:51:51.697023 5334 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260806 03:51:51.698042 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:51.686100 3542 leader_election.cc:334] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.506s (SENT)
I20260806 03:51:51.703083 3542 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:51.677485 5248 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:51:51.711560 2332 leader_election.cc:334] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 3.154s (SENT)
I20260806 03:51:51.721951 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.733115 2575 raft_consensus.cc:2396] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:51.736625 4906 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:51.737298 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.686615 5141 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.747293 2583 raft_consensus.cc:1226] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:51:51.748009 2572 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:51.684487 5320 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:51.753449 5245 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:51.749187 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.766175 5113 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:51.706928 5130 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
I20260806 03:51:51.774060 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:51.795703 2369 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:51.780645 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.777676 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.781433 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:51.799031 2575 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
W20260806 03:51:51.802484 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 91001.9us
I20260806 03:51:51.779487 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:51.769694 5245 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:51.788293 5320 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.776153 2368 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260806 03:51:51.784621 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 106074us
I20260806 03:51:51.790562 2374 raft_consensus.cc:2379] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:51.779278 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.773705 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:51.792218 5268 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.775328 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:51.803069 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 123900us
I20260806 03:51:51.781138 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:51.897850 2370 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:51:51.885525 2572 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:51:51.778515 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.778064 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.777252 5304 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:51.887173 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 294939us
I20260806 03:51:51.888240 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:51.909801 2579 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:51:51.923610 3593 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:51.932786 2372 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:51.924228 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:51.933707 5113 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:51:51.934469 3587 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:51.935928 2583 raft_consensus.cc:2379] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:51:51.936853 5032 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.593s user 0.013s sys 0.000s
I20260806 03:51:51.937672 5320 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:51.917308 5245 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.929069 2578 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:51.783833 5209 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:51.938735 2436 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.771926 5141 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.938902 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:51.939909 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:51.940562 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:51.970227 3593 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:51.940900 3547 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:51.941483 5097 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.941612 5218 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:51:51.900566 5278 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.799111 5119 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:51.984416 5268 consensus_queue.cc:1046] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:51:51.986078 5268 raft_consensus.cc:3032] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:51.947643 2577 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:51.922968 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:51.911525 5329 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:51.923142 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.953029 2534 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:51.937784 5001 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:51.964113 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 1983) took 5352 ms. Trace:
W20260806 03:51:51.975430 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 51317.1us
I20260806 03:51:51.978945 5209 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:51.980325 3594 raft_consensus.cc:2379] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:51.982041 2436 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:51.982590 5218 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:51.982668 5067 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:51.983125 3587 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:51:51.983675 5278 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.013705 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:52.031229 3542 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:52.233923 2377 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:51.987110 2581 raft_consensus.cc:1226] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:52.000489 2379 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:52.028839 2373 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260806 03:51:52.197932 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 216966us
I20260806 03:51:52.239655 2331 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.034605 2538 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:52.013661 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:52.241503 2376 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:52.028731 5141 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:52.242679 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:51.943249 5304 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:52.225559 3593 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:52.246537 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:52.247534 3593 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:51.911959 2474 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
W20260806 03:51:52.255241 2332 leader_election.cc:334] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.517s (SENT)
I20260806 03:51:52.268033 2575 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:52.268898 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:52.269470 2370 raft_consensus.cc:2396] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:51.986547 5268 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:51.986397 5119 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:52.272639 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 52126.9us
W20260806 03:51:52.278234 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 371223us
I20260806 03:51:52.248390 5236 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:52.280973 2377 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:52.028306 5281 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:52.254487 5097 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:51.996495 5320 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:52.235903 2481 rpcz_store.cc:276] 0806 03:51:46.598449 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.306868 (+4708419us) service_pool.cc:227] Handling call
0806 03:51:51.531178 (+224310us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65bce 000000000034aea9 00007f0d81991a30 00007f0d81a577b4 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:51:51.569025 (+ 37847us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:51.569779 (+ 754us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:51.946044 (+376265us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b54001abb50. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:51.950961 (+ 4917us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89674752}
I20260806 03:51:52.292645 2534 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:52.292464 5329 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:52.297819 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:52.018715 5095 consensus_queue.cc:1046] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.271s
W20260806 03:51:52.298043 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 316133us
I20260806 03:51:52.298852 5268 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:52.300427 5268 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:52.300681 2436 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:52.302670 2436 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:52.304044 5303 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:52.304812 5303 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:52.306531 2436 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:52.284762 4993 consensus_meta.cc:287] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.663s user 0.013s sys 0.002s
W20260806 03:51:52.311715 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 342219us
I20260806 03:51:52.314517 2538 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:52.315536 5236 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.315618 5067 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:52.316939 4983 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:52.317718 5119 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:52.296775 2537 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:52.334613 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:52.340723 2478 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:52.340499 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 53733.5us
W20260806 03:51:52.341514 2332 leader_election.cc:334] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.587s (SENT)
I20260806 03:51:52.281778 5141 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:51:52.342232 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 102771us
W20260806 03:51:52.349992 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 53745.8us
I20260806 03:51:52.289750 5248 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.358590 2429 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.373358 2331 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.374600 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.376353 4993 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.377089 4993 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.377583 4993 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:52.376480 5095 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:52.375756 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.323897 5067 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:51:52.380334 2332 leader_election.cc:334] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.706s (SENT)
I20260806 03:51:52.380927 2429 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.383340 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:52.364940 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:52.389958 5067 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:52.399827 5256 raft_consensus.cc:3032] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:51:52.400257 2481 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:52.400627 5256 raft_consensus.cc:729] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:52.404279 5385 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 03:51:52.406775 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 92333.8us
W20260806 03:51:52.407672 2332 leader_election.cc:334] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.890s (SENT)
I20260806 03:51:52.408098 2538 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
W20260806 03:51:52.408493 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 65473.3us
I20260806 03:51:52.408324 2332 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:52.289594 5334 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.079s
I20260806 03:51:52.364483 5116 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
W20260806 03:51:52.416110 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 54240.6us
I20260806 03:51:52.378592 5001 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:52.415268 5130 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:52.437865 2436 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:52.440872 5304 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.443733 5304 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:52.347312 5097 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:52.429039 5303 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.441532 5141 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.421499 2534 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:52.390426 5329 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.366791 5248 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.386596 5095 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.388520 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:52.327960 5320 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.416853 4983 raft_consensus.cc:686] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:52.402056 5256 consensus_queue.cc:260] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:52.463223 2369 consensus_meta.cc:287] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.562s user 0.013s sys 0.000s
I20260806 03:51:52.463106 2534 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:52.477867 2369 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:51:52.456322 2578 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
W20260806 03:51:52.484272 2235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33762 (request call id 224) took 13215 ms (client timeout 15000 ms). Trace:
I20260806 03:51:52.497133 2478 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:52.462437 4978 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:52.463287 5329 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.467603 5095 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:52.511314 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 518965us
I20260806 03:51:52.461087 5141 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:52.472702 5387 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:52.482168 2236 catalog_manager.cc:5495] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 3 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:52.459642 5303 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.463812 5248 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:52.463224 5116 raft_consensus.cc:686] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:52.479878 5032 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:52.504680 5256 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:52.475566 2474 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:52.516083 5097 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:52.521119 4978 raft_consensus.cc:686] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:52.517397 5245 consensus_meta.cc:287] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.506s user 0.015s sys 0.000s
W20260806 03:51:52.529259 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 121373us
I20260806 03:51:52.522112 2236 catalog_manager.cc:5495] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:52.523947 3587 raft_consensus.cc:2396] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:52.518200 5329 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:52.519906 5334 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.522516 5303 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:52.517163 5387 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.517664 5278 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:52.525498 5032 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:52.484858 2235 rpcz_store.cc:269] 0806 03:51:39.268704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:39.269612 (+ 908us) service_pool.cc:227] Handling call
0806 03:51:52.484171 (+13214559us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:51.269635 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:51.270729 (+ 1094us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:51.270804 (+ 75us) write_op.cc:435] Acquired schema lock
0806 03:51:51.270860 (+ 56us) tablet.cc:660] Decoding operations
0806 03:51:51.271645 (+ 785us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:51.271784 (+ 139us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:51.271827 (+ 43us) tablet.cc:683] Acquiring locks for 2 operations
0806 03:51:51.343968 (+ 72141us) tablet.cc:699] Row locks acquired
0806 03:51:51.347638 (+ 3670us) write_op.cc:260] PREPARE: finished
0806 03:51:51.385852 (+ 38214us) write_op.cc:270] Start()
0806 03:51:51.408728 (+ 22876us) write_op.cc:276] Timestamp: P: 1785988311359722 usec, L: 0
0806 03:51:51.408784 (+ 56us) op_driver.cc:331] REPLICATION: starting
0806 03:51:51.409761 (+ 977us) log.cc:842] Serialized 917 byte log entry
0806 03:51:51.775771 (+366010us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b1800365a40. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d764 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:51:51.926290 (+150519us) op_driver.cc:444] REPLICATION: finished
0806 03:51:52.376970 (+450680us) write_op.cc:301] APPLY: starting
0806 03:51:52.377188 (+ 218us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:52.377394 (+ 206us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:52.377429 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:52.433850 (+ 56421us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:52.434127 (+ 277us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0806 03:51:52.434173 (+ 46us) write_op.cc:312] APPLY: finished
0806 03:51:52.434533 (+ 360us) log.cc:842] Serialized 29 byte log entry
0806 03:51:52.434745 (+ 212us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:52.434838 (+ 93us) write_op.cc:454] Released schema lock
0806 03:51:52.434904 (+ 66us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:52.436200 (+ 1296us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:52.436227 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3917393,"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":446279,"apply.run_cpu_time_us":3785,"apply.run_wall_time_us":59819,"num_ops":2,"prepare.queue_time_us":113894,"prepare.run_cpu_time_us":31650,"prepare.run_wall_time_us":215215,"raft.queue_time_us":335142,"raft.run_cpu_time_us":49222,"raft.run_wall_time_us":381619,"replication_time_us":516800,"spinlock_wait_cycles":10040064,"thread_start_us":613651,"threads_started":4,"wal-append.queue_time_us":92956}]]}
I20260806 03:51:52.577629 2236 catalog_manager.cc:5495] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 3, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 3 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 03:51:52.592561 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 70183us
I20260806 03:51:52.571201 5334 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.602705 4978 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:52.598793 5278 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:52.604939 5278 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:52.540547 5245 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.608319 2236 catalog_manager.cc:5495] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 3 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:52.609774 5245 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:52.610342 2236 catalog_manager.cc:5495] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:52.611152 3542 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:52.605777 5278 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.579496 5387 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:52.583148 5032 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:52.584378 5236 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:52.624465 2236 catalog_manager.cc:5495] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 3, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 3 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:52.601913 5334 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:52.630119 5113 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:51:52.631587 5380 log.cc:924] Time spent T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.094s user 0.000s sys 0.001s
W20260806 03:51:52.632380 5373 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.216s user 0.000s sys 0.001s
I20260806 03:51:52.633394 5278 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:52.656128 5377 log.cc:924] Time spent T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20260806 03:51:52.626807 5097 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:51:52.638828 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 107038us
I20260806 03:51:52.623514 4983 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:52.652210 5268 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:52.638118 5320 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:52.637192 5236 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.666815 2236 catalog_manager.cc:5495] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 3, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:51:52.682076 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 306380us
I20260806 03:51:52.682855 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.661516 5218 raft_consensus.cc:686] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:52.647626 5067 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:52.691133 5127 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.011s sys 0.026s
W20260806 03:51:52.694904 5127 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s user 0.012s sys 0.026s
I20260806 03:51:52.691135 2236 catalog_manager.cc:5495] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 3, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 3 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:52.696336 2236 catalog_manager.cc:5495] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 2 to 4, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 4 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:51:52.704118 2331 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.678493 5236 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:52.707835 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 327160us
I20260806 03:51:52.681432 5320 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:51:52.728082 3542 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:52.698508 2236 catalog_manager.cc:5495] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:52.636453 5116 consensus_queue.cc:237] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.664714 4993 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:52.677871 5268 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.715130 5236 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:52.748476 3547 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:52.760258 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:52.763248 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:52.763566 2578 raft_consensus.cc:3032] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:51:52.763396 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 51701.3us
I20260806 03:51:52.764138 2578 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:52.766747 3547 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:52.768150 2538 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:52.768524 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:52.769569 2575 raft_consensus.cc:2396] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:52.770061 5209 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:52.769990 2538 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:52.771103 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:52.772194 3547 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:52.742343 5320 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:52.768173 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:52.773622 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 265794us
I20260806 03:51:52.781771 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:52.782137 2534 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:52.782616 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 374994us
I20260806 03:51:52.783540 2429 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:52.754302 5388 raft_consensus.cc:2781] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:52.786785 5388 raft_consensus.cc:686] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:52.786795 2538 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:52.788887 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:52.789899 2579 raft_consensus.cc:2396] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:51:52.790666 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:52.786110 5209 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:51:52.768196 2578 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:52.794790 2581 raft_consensus.cc:2354] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:52.738452 5278 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260806 03:51:52.796810 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:52.799283 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:52.800547 2578 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:52.741909 5329 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.805567 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:52.759685 5245 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:52.750917 5304 consensus_queue.cc:1046] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.034s
I20260806 03:51:52.803032 3594 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:52.804149 2534 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:52.776876 5113 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:52.772964 5390 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:52.796164 5097 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:51:52.863821 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 77100.7us
W20260806 03:51:52.864008 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 59980.6us
I20260806 03:51:52.802264 5001 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.788304 5388 consensus_queue.cc:237] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:52.864984 2534 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:52.885205 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 45443.1us
W20260806 03:51:52.887391 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 449660us
I20260806 03:51:52.813838 4978 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.818434 5390 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.791960 5320 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.825279 5097 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.873905 5001 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:51:52.873335 5376 log.cc:924] Time spent T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
W20260806 03:51:52.907846 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 119897us
I20260806 03:51:52.921154 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:52.817835 5245 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:52.936079 2368 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:52.869027 5113 raft_consensus.cc:686] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:52.904831 4978 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:52.942569 2370 raft_consensus.cc:1226] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: [2.4-2.4]
I20260806 03:51:52.943498 2578 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:52.952939 2379 raft_consensus.cc:1261] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:51:52.904604 5392 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:52.912243 5127 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:52.956177 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:52.914799 5130 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:51:52.912761 5390 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:52.929690 5097 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:52.811546 5218 consensus_queue.cc:237] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:52.928925 5001 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:52.960919 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 176332us
W20260806 03:51:52.977002 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 68595.8us
I20260806 03:51:52.959260 4978 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:53.016057 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:53.006969 5304 raft_consensus.cc:3032] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:51:53.032079 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:52.945783 5067 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.007364 4993 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:52.961227 5392 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:53.055040 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.014704 5268 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.071334 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:53.086623 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:53.036568 5304 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:53.054512 5393 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:53.091744 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:53.093678 5399 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:53.094261 5399 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:53.092743 2368 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:53.015368 5095 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.094694 5399 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:53.098197 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.067600 5392 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:53.103206 5384 log.cc:924] Time spent T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.616s user 0.004s sys 0.004s
W20260806 03:51:53.103703 5391 log.cc:924] Time spent T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.216s user 0.000s sys 0.000s
I20260806 03:51:53.104247 5407 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.109484 5113 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.111199 5268 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:53.116281 2331 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
W20260806 03:51:53.116632 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 64294.4us
W20260806 03:51:53.093334 5248 consensus_meta.cc:287] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.556s user 0.011s sys 0.004s
W20260806 03:51:53.145028 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 136655us
I20260806 03:51:53.113515 5390 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.096691 4993 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:53.100571 5329 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.136291 5393 raft_consensus.cc:686] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:53.161185 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.174691 2368 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 5
W20260806 03:51:53.171347 5303 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.583s user 0.010s sys 0.002s
I20260806 03:51:53.189180 2436 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.101226 5334 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.191931 5303 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.148782 5127 raft_consensus.cc:686] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:53.210312 5303 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.214636 5130 raft_consensus.cc:686] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:53.157517 5429 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:53.204231 5256 consensus_meta.cc:287] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.677s user 0.009s sys 0.005s
I20260806 03:51:53.154709 5268 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.199043 5400 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:53.268360 2577 raft_consensus.cc:2396] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260806 03:51:53.271283 5430 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.221156 5329 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:53.224895 5032 consensus_meta.cc:287] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.568s user 0.011s sys 0.002s
W20260806 03:51:53.275786 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 102656us
I20260806 03:51:53.172834 4993 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.226184 5320 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.158182 5248 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.294641 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1033) took 1318 ms. Trace:
I20260806 03:51:53.295212 2375 rpcz_store.cc:276] 0806 03:51:51.975953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:51.976188 (+ 235us) service_pool.cc:227] Handling call
0806 03:51:51.976536 (+ 348us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:51.976995 (+ 459us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:51.977039 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:51.977570 (+ 531us) log.cc:842] Serialized 26 byte log entry
0806 03:51:52.252734 (+275164us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:52.252801 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:52.252919 (+ 118us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:52.957591 (+704672us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:53.280945 (+323354us) raft_consensus.cc:1671] finished
0806 03:51:53.280978 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:53.294550 (+ 13572us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1533,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":31729536,"thread_start_us":274657,"threads_started":1,"wal-append.queue_time_us":470270}
I20260806 03:51:53.300200 2436 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.255147 5245 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:53.308394 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:53.162695 5407 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.184662 5304 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.309906 4983 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.287329 5032 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:53.363647 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 63565.2us
W20260806 03:51:53.364614 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 248437us
I20260806 03:51:53.301420 5434 consensus_queue.cc:1035] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.083s
I20260806 03:51:53.372430 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:51:53.372776 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 357938us
I20260806 03:51:53.373281 2474 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:53.375174 5415 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.376695 2331 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.256253 5429 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.340916 5329 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:51:53.415512 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.345350 5095 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:53.416564 5434 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:53.421483 2436 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:53.417163 5248 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:53.297411 5130 consensus_queue.cc:237] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.362138 5303 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.286755 5430 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.344623 5438 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:53.365646 5304 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:53.428871 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:53.418457 5390 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.330746 5400 consensus_queue.cc:1035] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.102s
I20260806 03:51:53.438989 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:53.341600 5320 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:53.417907 5432 consensus_queue.cc:1035] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260806 03:51:53.448551 5425 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.449268 5425 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.453099 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:53.453919 3587 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:53.454643 5141 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.926s user 0.015s sys 0.001s
I20260806 03:51:53.455117 5141 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.456708 3594 raft_consensus.cc:1226] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
W20260806 03:51:53.457791 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 43497.1us
I20260806 03:51:53.458045 2370 raft_consensus.cc:2379] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:53.414170 5415 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.459306 2436 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.361370 5127 consensus_queue.cc:237] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.325918 5393 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.462131 4978 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:53.463698 5444 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260806 03:51:53.457367 5141 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:53.464393 5268 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.459318 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:53.441351 5398 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260806 03:51:53.465657 5408 log.cc:924] Time spent T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.245s user 0.000s sys 0.001s
I20260806 03:51:53.472093 2376 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:51:53.477789 5403 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:53.477231 2373 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:53.479876 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.451872 5425 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.496286 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:53.490686 2332 leader_election.cc:334] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.959s (SENT)
I20260806 03:51:53.497231 2332 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.492350 2369 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:53.499221 2577 raft_consensus.cc:2379] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:53.414147 4983 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.448570 5439 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.441653 5400 consensus_queue.cc:1225] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer dff94100ffc5485f92cbe6a85addf0b2 log is divergent from this leader: its last log entry 4.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 03:51:53.467123 5334 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:53.436333 5303 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.521955 5403 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.490907 5398 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:53.457415 5429 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:53.524828 4993 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.708145 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.755821 2373 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:53.519311 5435 consensus_queue.cc:1046] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.017s
I20260806 03:51:53.736884 4978 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:53.441154 5438 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:53.760291 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 301135us
I20260806 03:51:53.760875 5438 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.761380 2436 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.761472 5415 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:53.763216 2436 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:53.696842 2377 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s user 0.026s sys 0.029s
I20260806 03:51:53.718294 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:53.766685 2377 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s user 0.029s sys 0.030s
I20260806 03:51:53.523985 5437 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260806 03:51:53.515319 5434 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:51:53.726441 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 350990us
I20260806 03:51:53.503578 5268 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:53.501770 5320 raft_consensus.cc:686] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:53.773671 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.775668 2372 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:53.729965 5116 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.744443 5398 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:53.797883 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 324223us
I20260806 03:51:53.822861 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:53.764155 5278 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260806 03:51:53.826321 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 53317.4us
W20260806 03:51:53.831162 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 100388us
I20260806 03:51:53.770366 5067 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:53.836833 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.764743 5407 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:53.790827 5245 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:53.845768 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 430329us
I20260806 03:51:53.795792 5032 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:53.847293 2478 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:51:53.848985 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:53.849179 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.851617 2376 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:53.851611 3593 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:53.839926 2377 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:53.834676 5435 consensus_queue.cc:1225] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer cf6042e85bef47c4aac9d064d91501d1 log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 03:51:53.853392 2369 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:53.515120 5439 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:53.873036 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 109875us
I20260806 03:51:53.853971 2331 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:53.875315 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:53.880319 2537 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:53.841635 4993 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:53.858464 5401 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.026s sys 0.039s
I20260806 03:51:53.927011 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.936200 4978 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:53.936717 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:53.937568 2369 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:51:53.934006 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 103461us
W20260806 03:51:53.934664 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 54460.3us
I20260806 03:51:53.878329 5393 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:53.950088 5429 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.955574 5393 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.956374 5429 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:53.957216 5454 log.cc:924] Time spent T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
I20260806 03:51:53.958590 5429 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.959147 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.959805 2379 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:53.945622 2368 consensus_meta.cc:287] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.741s user 0.011s sys 0.000s
I20260806 03:51:53.960893 2368 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:53.961306 5399 consensus_meta.cc:287] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.808s user 0.010s sys 0.003s
I20260806 03:51:53.961737 5399 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:53.949820 5097 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.925s user 0.016s sys 0.010s
I20260806 03:51:53.962232 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.963153 2369 raft_consensus.cc:2379] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:53.962860 5097 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:53.964784 5456 log.cc:924] Time spent T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.209s user 0.001s sys 0.000s
I20260806 03:51:53.965442 2534 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:53.962340 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:53.968729 5430 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:53.969408 5430 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:53.969889 5430 raft_consensus.cc:686] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:53.970798 5430 consensus_queue.cc:237] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.952381 5032 consensus_queue.cc:1035] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260806 03:51:53.973192 2577 raft_consensus.cc:3032] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:51:53.973408 5097 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:53.954016 4993 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.974895 5113 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.975468 5113 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.975857 4993 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.973629 2577 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:53.977151 2577 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.977689 5113 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.978384 2577 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:53.979199 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:53.979663 2479 raft_consensus.cc:1226] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:53.981201 2481 raft_consensus.cc:1203] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Deduplicated request from leader. Original: 1.1->[2.2-2.2] Dedup: 2.2->[]
I20260806 03:51:53.981251 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:53.981954 2481 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260806 03:51:53.938968 5401 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.719s user 0.031s sys 0.043s
I20260806 03:51:53.936694 4978 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:53.983108 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:53.957217 5320 consensus_queue.cc:237] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.985726 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:53.986440 2575 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:53.987636 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:53.988209 2436 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.989787 2436 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:53.989486 5097 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.990540 5097 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.831475 5116 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:53.992303 5329 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:53.992975 5329 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:53.994123 5116 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:53.995544 5412 log.cc:924] Time spent T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.709s user 0.003s sys 0.000s
I20260806 03:51:53.952102 5469 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:53.997159 5469 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:53.999609 5469 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:54.000324 3545 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:54.000706 5001 consensus_meta.cc:287] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.972s user 0.016s sys 0.011s
I20260806 03:51:54.001235 5001 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.000949 5097 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.003494 2578 raft_consensus.cc:1226] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:54.003702 5097 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:54.004237 5097 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:54.004709 5236 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.240s user 0.012s sys 0.004s
I20260806 03:51:54.005375 5236 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.007073 2374 raft_consensus.cc:1226] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
W20260806 03:51:54.008906 5392 consensus_meta.cc:287] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.861s user 0.012s sys 0.001s
I20260806 03:51:54.009464 5392 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.009950 5439 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.010382 5303 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:53.983335 4978 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:54.012233 5387 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.376s user 0.014s sys 0.004s
W20260806 03:51:54.012961 5455 log.cc:924] Time spent T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.271s user 0.000s sys 0.003s
I20260806 03:51:54.012730 5387 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.014689 5334 raft_consensus.cc:3032] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:51:54.014799 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:54.015159 5334 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:54.015589 2575 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:53.854480 5245 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.016166 5334 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.017416 5392 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.017522 5334 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:54.017868 5245 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:54.018918 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:53.951299 2332 leader_election.cc:334] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.088s (SENT)
I20260806 03:51:54.019944 2369 raft_consensus.cc:1226] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260806 03:51:53.929868 2429 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:54.020838 2581 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
W20260806 03:51:54.020793 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 91075.8us
I20260806 03:51:54.021283 2581 pending_rounds.cc:76] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 3
I20260806 03:51:54.027148 2436 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:54.032994 2373 raft_consensus.cc:2396] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:54.004652 5097 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:53.996609 2577 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260806 03:51:54.052503 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:54.054522 2577 pending_rounds.cc:76] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 6
I20260806 03:51:54.015065 5387 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.021528 2581 pending_rounds.cc:97] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20260806 03:51:54.054903 2577 pending_rounds.cc:97] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260806 03:51:53.958946 5393 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.064137 2581 raft_consensus.cc:2866] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:54.064137 2577 raft_consensus.cc:2866] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:54.071321 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:54.072873 2370 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:54.064808 2376 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:54.076385 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.052518 5218 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:54.166196 2373 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:54.056188 5392 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:54.169351 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 152) took 1857 ms. Trace:
I20260806 03:51:54.215812 2534 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.167799 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:54.170715 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2088) took 1409 ms. Trace:
W20260806 03:51:54.177560 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 331027us
I20260806 03:51:54.170991 2379 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:54.220752 2331 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:54.213070 2482 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:51:54.079481 5303 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:51:54.215655 2476 rpcz_store.cc:276] 0806 03:51:52.312008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:52.334948 (+ 22940us) service_pool.cc:227] Handling call
0806 03:51:52.365012 (+ 30064us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:52.754253 (+389241us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:52.754304 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:52.775283 (+ 20979us) log.cc:842] Serialized 26 byte log entry
0806 03:51:53.028107 (+252824us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:51:53.028159 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:53.028245 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:53.994746 (+966501us) spinlock_profiling.cc:243] Waited 4.12 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:54.082548 (+ 87802us) raft_consensus.cc:1671] finished
0806 03:51:54.082583 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:54.169262 (+ 86679us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":45851,"raft.run_cpu_time_us":1505,"raft.run_wall_time_us":14184,"spinlock_wait_cycles":9513088,"thread_start_us":545348,"threads_started":2,"wal-append.queue_time_us":382586}
I20260806 03:51:54.224315 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.218158 2574 rpcz_store.cc:276] 0806 03:51:52.760843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:52.761119 (+ 276us) service_pool.cc:227] Handling call
0806 03:51:54.083298 (+1322179us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b5400156154. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:54.170630 (+ 87332us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1735268096}
I20260806 03:51:54.237747 2370 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:54.081708 5032 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:54.241324 2484 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:51:54.088372 5095 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:54.245255 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:54.246367 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 295093us
I20260806 03:51:54.245388 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.213548 5218 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:54.203025 5437 raft_consensus.cc:2741] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
I20260806 03:51:54.270900 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:54.248467 3587 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.794s user 0.014s sys 0.005s
I20260806 03:51:54.275231 2479 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260806 03:51:54.275665 2479 pending_rounds.cc:76] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 1
I20260806 03:51:54.275969 2479 pending_rounds.cc:97] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260806 03:51:54.276294 2479 raft_consensus.cc:2866] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:54.277401 2370 pending_rounds.cc:76] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 2
I20260806 03:51:54.277737 2370 pending_rounds.cc:97] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:51:54.277992 2370 raft_consensus.cc:2866] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:54.278806 2373 raft_consensus.cc:2379] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:54.261911 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.223958 5392 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:54.280185 2482 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.238278 5393 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:54.287477 2369 raft_consensus.cc:1261] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 03:51:54.288501 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 67830us
I20260806 03:51:54.289520 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:54.266194 2474 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.892s user 0.011s sys 0.003s
I20260806 03:51:54.204519 5469 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260806 03:51:54.243062 5435 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:51:54.308542 2429 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:54.310292 2478 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 03:51:54.339140 2482 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:54.341424 2331 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.272322 5218 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.347913 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.351440 2481 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.275065 4983 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:54.349342 2484 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:54.348714 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:54.352659 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 121559us
I20260806 03:51:54.288259 5392 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.251354 5032 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:54.356781 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 48393.6us
I20260806 03:51:54.317090 5435 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:54.341962 5388 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:54.373924 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 101998us
I20260806 03:51:54.362099 2534 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.284335 3587 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:54.368500 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 67498.1us
I20260806 03:51:54.233793 5471 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.135s
I20260806 03:51:54.344409 5405 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:54.345139 5067 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:54.300182 2474 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:51:54.351570 5329 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:54.268033 5095 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.345407 5393 raft_consensus.cc:686] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:54.379343 2429 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:51:54.372821 5304 consensus_meta.cc:287] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.931s user 0.013s sys 0.002s
I20260806 03:51:54.413861 5032 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:54.414577 5393 consensus_queue.cc:237] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.416597 2479 raft_consensus.cc:1226] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:54.417769 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.418473 2479 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:54.393043 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.376380 4993 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:54.398008 2331 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.419404 2484 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.419894 5392 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.422041 5392 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:54.402824 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 46465.2us
I20260806 03:51:54.406353 5445 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:54.422976 5445 raft_consensus.cc:686] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:54.424378 5464 consensus_queue.cc:1035] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:54.407480 4983 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:54.426663 4983 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.429183 4983 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.345721 5281 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260806 03:51:54.431038 5474 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:51:54.431537 5445 consensus_queue.cc:237] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.432215 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:54.433068 3593 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:54.410657 2538 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:54.411968 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:54.434623 2369 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:54.380108 5435 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:51:54.412063 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.436935 2480 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:54.437937 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.438609 2480 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.438265 2484 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:54.413024 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.373672 5268 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:54.440253 2478 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:54.439810 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.379276 5388 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.441576 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.444567 2537 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.389969 5396 consensus_queue.cc:1046] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260806 03:51:54.453197 3588 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:54.453509 5204 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:51:54.454018 5204 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:54.454288 5209 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:54.454546 5204 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:54.454916 5209 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:54.455387 5209 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.453881 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.460194 2436 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:54.388478 5236 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.442796 5388 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.472673 2369 raft_consensus.cc:1261] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:51:54.479526 2478 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:54.481266 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.419414 4993 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:54.471228 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:54.481994 5425 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:54.487703 5470 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.263s user 0.000s sys 0.001s
W20260806 03:51:54.487931 5467 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.247s user 0.001s sys 0.000s
I20260806 03:51:54.487567 5425 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:54.488224 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 110023us
I20260806 03:51:54.489352 5414 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:54.489881 5414 raft_consensus.cc:686] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:54.490054 5425 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:54.490993 5414 consensus_queue.cc:237] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.492619 5303 consensus_queue.cc:1046] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:51:54.474867 5116 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:51:54.493825 3547 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
W20260806 03:51:54.494483 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 48642.5us
I20260806 03:51:54.495062 5281 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:51:54.495086 5278 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:51:54.472980 5268 raft_consensus.cc:686] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:54.445245 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 156) took 1112 ms. Trace:
I20260806 03:51:54.514632 2481 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:54.470029 5401 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:51:54.516800 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 106287us
I20260806 03:51:54.517627 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.470185 5476 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:54.522590 2377 consensus_meta.cc:287] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.587s user 0.010s sys 0.003s
I20260806 03:51:54.512496 4993 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.545047 5476 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.473726 5399 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.399575 5067 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:54.550143 5398 consensus_meta.cc:287] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.712s user 0.012s sys 0.000s
I20260806 03:51:54.550710 5398 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.533006 2474 rpcz_store.cc:276] 0806 03:51:53.333333 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:53.344377 (+ 11044us) service_pool.cc:227] Handling call
0806 03:51:54.399756 (+1055379us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:54.552443 5067 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.445467 5329 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.553458 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.554248 2474 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:51:54.555258 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.555406 5329 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.556530 2482 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260806 03:51:54.558475 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.559404 2484 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:54.563484 5334 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.545s user 0.011s sys 0.002s
I20260806 03:51:54.535456 5268 consensus_queue.cc:237] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.564214 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.565191 2476 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:51:54.459528 4978 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.566449 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.530112 5396 raft_consensus.cc:3032] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:54.537130 5116 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.456773 5405 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.544745 2436 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:54.563208 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:54.530141 5425 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:54.573359 5396 raft_consensus.cc:729] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:54.591127 5425 raft_consensus.cc:686] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:54.579072 5476 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.591761 5396 consensus_queue.cc:260] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.570379 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.592423 5425 consensus_queue.cc:237] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.571802 3547 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:54.574591 2478 raft_consensus.cc:2396] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:54.531286 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.570011 2377 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:51:54.596405 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1794) took 1088 ms. Trace:
W20260806 03:51:54.589025 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 77526.7us
I20260806 03:51:54.598140 2481 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:54.593370 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:54.640898 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1769) took 1720 ms. Trace:
I20260806 03:51:54.659792 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1040) took 1330 ms. Trace:
I20260806 03:51:54.662804 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:54.663637 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 274480us
W20260806 03:51:54.676007 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 106587us
I20260806 03:51:54.692239 2372 rpcz_store.cc:276] 0806 03:51:53.329182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:53.507363 (+178181us) service_pool.cc:227] Handling call
0806 03:51:54.659522 (+1152159us) spinlock_profiling.cc:243] Waited 8.71 ms on lock 0x7b54003e9ed4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:54.659701 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20033792}
I20260806 03:51:54.633780 5304 raft_consensus.cc:3032] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:54.694974 5304 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:54.695309 5001 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.695757 5407 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:54.696137 5304 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.695112 5427 consensus_queue.cc:1035] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.002s
I20260806 03:51:54.697753 5304 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:54.633069 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.699957 2474 raft_consensus.cc:2379] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:54.601918 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.603235 3547 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:51:54.596863 2377 rpcz_store.cc:276] 0806 03:51:53.508106 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:53.771143 (+263037us) service_pool.cc:227] Handling call
0806 03:51:54.596293 (+825150us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:51:54.679246 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 134604us
I20260806 03:51:54.701283 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:54.693641 2374 rpcz_store.cc:276] 0806 03:51:52.920810 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:52.932229 (+ 11419us) service_pool.cc:227] Handling call
0806 03:51:52.977872 (+ 45643us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:54.007622 (+1029750us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:54.640538 (+632916us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54003331d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:54.640808 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2366315264}
W20260806 03:51:54.600793 3542 leader_election.cc:334] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.902s (SENT)
I20260806 03:51:54.704092 3542 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:54.701375 2479 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:51:54.702497 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:54.705082 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 101954us
I20260806 03:51:54.664389 5465 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:54.675103 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1048) took 1221 ms. Trace:
I20260806 03:51:54.699648 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:54.673455 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:54.714185 2484 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:51:54.763274 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 49449.9us
W20260806 03:51:54.625391 5097 consensus_meta.cc:287] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.558s user 0.012s sys 0.000s
I20260806 03:51:54.766469 2480 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:51:54.767856 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 66718.8us
W20260806 03:51:54.769627 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 87576.9us
I20260806 03:51:54.773350 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:54.579245 4978 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.770617 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:54.777875 2483 raft_consensus.cc:2379] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:54.807201 2374 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:54.777792 2479 raft_consensus.cc:1226] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:51:54.806098 2481 raft_consensus.cc:1261] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:51:54.785151 2377 raft_consensus.cc:2396] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:54.776993 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:54.809813 2372 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.714252 3587 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.763345 2368 rpcz_store.cc:276] 0806 03:51:53.453711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.215070 (+761359us) service_pool.cc:227] Handling call
0806 03:51:54.235309 (+ 20239us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:54.235618 (+ 309us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:54.460828 (+225210us) log.cc:842] Serialized 14 byte log entry
0806 03:51:54.674682 (+213854us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:54.674747 (+ 65us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:54.674790 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:54.674880 (+ 90us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:54.675034 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":224512,"prepare.queue_time_us":161752,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":221322,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":112,"thread_start_us":131383,"threads_started":2,"wal-append.queue_time_us":72218}
I20260806 03:51:54.601771 5329 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:54.624537 5116 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:54.636394 5281 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:51:54.715548 5474 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:51:54.813705 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 48564.5us
I20260806 03:51:54.840943 2331 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.813623 3547 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:54.713584 5396 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:54.661747 5469 consensus_queue.cc:1046] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.115s
I20260806 03:51:54.643334 5398 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.824452 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:54.825165 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:54.851174 2538 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:54.896885 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:54.828305 5281 raft_consensus.cc:686] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:54.834071 5435 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.642473 5209 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.838063 2372 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:54.592459 5405 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:54.848862 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 248124us
I20260806 03:51:54.636343 5403 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.849206 2331 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.660571 5218 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.767287 5393 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:54.849612 2484 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:51:54.891939 2474 raft_consensus.cc:1261] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:51:54.817484 2436 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:54.776888 5415 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:54.888536 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:54.777530 5385 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:54.895404 5119 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:54.849573 5474 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:51:54.918502 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 154097us
W20260806 03:51:54.922362 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 73517.7us
I20260806 03:51:54.887686 5434 raft_consensus.cc:2741] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
W20260806 03:51:54.940419 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 123038us
I20260806 03:51:54.718022 5001 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.131s
W20260806 03:51:54.928306 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 77239.3us
I20260806 03:51:54.807368 5097 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:54.936198 2534 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.936313 2331 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:54.932770 5320 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:54.949533 2538 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:54.889497 5387 consensus_queue.cc:1046] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.045s
I20260806 03:51:54.810961 5204 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:54.939720 5496 log.cc:924] Time spent T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
W20260806 03:51:54.941459 5458 log.cc:924] Time spent T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 1.198s user 0.000s sys 0.008s
I20260806 03:51:54.939652 5474 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:54.939805 5403 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:54.977700 2436 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:54.951622 5405 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:51:54.814700 5465 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.940966 5218 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:54.976611 5434 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:54.993053 5405 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:54.957753 5415 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:51:54.954860 5320 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:54.935060 5393 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:54.985317 2538 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:54.998299 5209 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:54.940490 5281 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:55.015095 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 164410us
I20260806 03:51:55.017212 2538 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.010154 5405 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:51:55.004957 5398 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:55.003566 5434 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:55.003312 5256 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:55.050459 2538 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.033298 2436 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:55.056529 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 82080.1us
I20260806 03:51:55.001101 5403 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:55.010921 5320 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:55.066406 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:55.067425 2374 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:54.999979 5499 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20260806 03:51:55.015868 5415 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:51:55.057483 4993 consensus_meta.cc:287] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.507s user 0.010s sys 0.002s
W20260806 03:51:55.061007 5441 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20260806 03:51:55.094568 2429 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:55.010776 5387 consensus_queue.cc:1046] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.080s
I20260806 03:51:55.084228 2483 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:51:55.053136 5001 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:51:55.060081 5434 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:55.138375 5268 raft_consensus.cc:2741] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
W20260806 03:51:55.132403 5457 log.cc:924] Time spent T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 1.201s user 0.003s sys 0.005s
I20260806 03:51:55.144474 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 188) took 2059 ms. Trace:
I20260806 03:51:55.144935 2375 rpcz_store.cc:276] 0806 03:51:53.085385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:53.332560 (+247175us) service_pool.cc:227] Handling call
0806 03:51:53.362272 (+ 29712us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:51:53.417266 (+ 54994us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:53.417303 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:51:53.417942 (+ 639us) log.cc:842] Serialized 52 byte log entry
0806 03:51:53.438119 (+ 20177us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:51:53.506094 (+ 67975us) log.cc:842] Serialized 14 byte log entry
0806 03:51:53.514473 (+ 8379us) log.cc:842] Serialized 14 byte log entry
0806 03:51:53.514637 (+ 164us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:53.514711 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:55.131687 (+1616976us) spinlock_profiling.cc:243] Waited 5.29 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:55.144211 (+ 12524us) raft_consensus.cc:1671] finished
0806 03:51:55.144239 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:55.144401 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":106114,"prepare.queue_time_us":317318,"prepare.run_cpu_time_us":8835,"prepare.run_wall_time_us":198657,"raft.queue_time_us":78920,"raft.run_cpu_time_us":502,"raft.run_wall_time_us":9566,"spinlock_wait_cycles":12254336,"thread_start_us":19414,"threads_started":1,"wal-append.queue_time_us":356891}
I20260806 03:51:55.148656 2331 leader_election.cc:398] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:55.057035 5256 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:55.103642 5097 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:55.104310 5417 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:51:55.108913 5204 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:55.112939 5465 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:55.115437 5304 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:55.122918 5393 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:55.093631 5500 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:55.127285 5435 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:55.093578 4993 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:55.135591 5502 log.cc:924] Time spent T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.155s user 0.002s sys 0.000s
I20260806 03:51:55.136600 5403 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:55.149187 2331 leader_election.cc:401] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:51:55.102946 2538 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.098644 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:55.178920 2374 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:55.164291 5256 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:55.185796 5417 raft_consensus.cc:686] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:55.188709 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 155508us
W20260806 03:51:55.199625 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 96769.5us
W20260806 03:51:55.202850 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 112585us
I20260806 03:51:55.204409 2370 raft_consensus.cc:2379] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:55.211586 2538 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.185595 5469 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:55.215852 2331 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.230253 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:55.231462 3588 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:55.212219 5500 raft_consensus.cc:686] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:55.235460 5492 log.cc:924] Time spent T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.297s user 0.000s sys 0.001s
I20260806 03:51:55.236866 2436 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.238157 2538 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.208374 5403 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
W20260806 03:51:55.249825 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 313710us
I20260806 03:51:55.252439 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.256910 2538 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.259498 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:55.262075 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:55.262933 3592 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:51:55.262998 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:55.263659 3590 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:55.265429 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.259753 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:55.263921 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:55.276916 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:55.213850 5511 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:55.282807 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:55.278182 2372 raft_consensus.cc:2379] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:55.283736 3594 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:55.224676 5387 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:51:55.281505 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 59667.8us
I20260806 03:51:55.287026 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:55.292738 2373 raft_consensus.cc:2379] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:55.293872 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:55.209782 5304 consensus_queue.cc:1035] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.056s
I20260806 03:51:55.288733 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.304034 2373 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:55.305325 2376 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 03:51:55.306391 2374 raft_consensus.cc:1261] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:55.253427 5245 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:55.308733 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:55.309764 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:55.308302 3584 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:51:55.316320 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:55.317087 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:55.317641 5410 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.353s user 0.029s sys 0.020s
W20260806 03:51:55.317773 5515 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260806 03:51:55.332506 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.318826 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.274417 5334 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:55.318063 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:55.273770 5398 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:55.323931 2575 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:55.276822 5393 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:55.233412 5417 consensus_queue.cc:237] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:55.328228 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 71429.1us
I20260806 03:51:55.329548 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.333274 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:55.334115 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 76492.4us
I20260806 03:51:55.336148 3542 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.347519 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:55.348493 2374 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:51:55.348819 3542 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.344410 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.234939 5500 consensus_queue.cc:237] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:55.350533 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:55.351764 2372 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 4: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:55.267428 5469 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:55.326294 5414 consensus_queue.cc:1035] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.034s
I20260806 03:51:55.326858 5245 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:55.360801 3591 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: []
W20260806 03:51:55.329780 5410 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.446s user 0.029s sys 0.020s
I20260806 03:51:55.322578 5329 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:55.362398 2572 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:55.366055 5434 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:51:55.369446 5236 raft_consensus.cc:3032] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 03:51:55.370898 5495 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.357s user 0.001s sys 0.001s
I20260806 03:51:55.371618 5385 raft_consensus.cc:3032] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:55.377457 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:55.377538 2332 leader_election.cc:334] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.514s (SENT)
I20260806 03:51:55.416508 2332 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.327476 5510 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:55.422752 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 45259.3us
I20260806 03:51:55.391165 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:55.388589 3590 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
W20260806 03:51:55.397193 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 82255.9us
I20260806 03:51:55.401932 2583 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
W20260806 03:51:55.405033 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 50920.8us
I20260806 03:51:55.407055 2331 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:55.408576 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.409370 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.356868 5127 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:55.412314 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.418854 2579 raft_consensus.cc:1261] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:51:55.429371 3594 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:55.395955 5113 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:55.427953 3542 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.396417 5518 consensus_queue.cc:1046] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.167s
I20260806 03:51:55.434461 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:55.510493 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:55.622054 2576 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:55.483613 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 135749us
I20260806 03:51:55.623939 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 171) took 1047 ms. Trace:
I20260806 03:51:55.624378 2483 rpcz_store.cc:276] 0806 03:51:54.576472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.817343 (+240871us) service_pool.cc:227] Handling call
0806 03:51:54.905105 (+ 87762us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:51:54.919460 (+ 14355us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:54.919493 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:51:54.954610 (+ 35117us) log.cc:842] Serialized 574 byte log entry
0806 03:51:55.084048 (+129438us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:55.623275 (+539227us) log.cc:842] Serialized 14 byte log entry
0806 03:51:55.623564 (+ 289us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:55.623677 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:55.623719 (+ 42us) raft_consensus.cc:1671] finished
0806 03:51:55.623730 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:55.623876 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":10683,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":15997,"thread_start_us":94909,"threads_started":1,"wal-append.queue_time_us":224376}
W20260806 03:51:55.626718 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 198918us
I20260806 03:51:55.489070 2436 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.435623 2538 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:55.625489 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.632571 5097 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:51:55.632298 5393 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260806 03:51:55.634303 5539 log.cc:924] Time spent T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.215s user 0.000s sys 0.001s
I20260806 03:51:55.634608 3542 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.415553 5385 raft_consensus.cc:729] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
W20260806 03:51:55.634968 5525 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.217s user 0.001s sys 0.000s
I20260806 03:51:55.498988 2577 raft_consensus.cc:1226] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:55.644403 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 208916us
W20260806 03:51:55.596756 2332 leader_election.cc:334] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.726s (SENT)
I20260806 03:51:55.647207 2332 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.512835 2575 raft_consensus.cc:1226] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:51:55.417819 4993 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:55.569563 5530 consensus_queue.cc:1035] T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.083s
I20260806 03:51:55.417680 3586 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
W20260806 03:51:55.654377 5535 log.cc:924] Time spent T 2b2fa42f901d4b37bbfd3baccb897807 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.234s user 0.000s sys 0.001s
I20260806 03:51:55.654882 5095 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:55.657853 5528 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:51:55.642081 5471 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:55.664601 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1774) took 1255 ms. Trace:
I20260806 03:51:55.641558 5437 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:51:55.432106 5430 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:55.668445 5434 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:55.669019 2534 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:55.664987 5245 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:55.413827 5236 raft_consensus.cc:729] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:55.499017 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:55.665038 2478 rpcz_store.cc:276] 0806 03:51:54.409363 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.480968 (+ 71605us) service_pool.cc:227] Handling call
0806 03:51:55.664355 (+1183387us) spinlock_profiling.cc:243] Waited 91.5 ms on lock 0x7b54002c1854. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:55.664532 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":210506624}
I20260806 03:51:55.670048 5435 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:55.671260 2534 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:51:55.425693 3587 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.528s user 0.013s sys 0.001s
I20260806 03:51:55.678758 2583 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:55.679072 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 82371us
I20260806 03:51:55.679250 2574 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:51:55.683975 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:55.424271 5403 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260806 03:51:55.689705 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 200790us
I20260806 03:51:55.415680 5329 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:55.704756 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:55.569485 5469 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:55.698149 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:55.718758 2484 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:55.721199 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:55.722055 2474 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:51:55.698565 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:55.699574 2572 raft_consensus.cc:1226] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:55.701817 2478 raft_consensus.cc:1261] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 03:51:55.714609 2332 leader_election.cc:334] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.807s (SENT)
I20260806 03:51:55.473694 5517 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:55.481647 5465 raft_consensus.cc:2781] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:51:55.717872 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:55.651665 5393 consensus_queue.cc:1046] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260806 03:51:55.797931 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:55.685315 2483 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:55.726480 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:55.566684 3588 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:55.754168 2332 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.717434 2481 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:51:55.804301 2479 raft_consensus.cc:2379] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:55.768534 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:55.676313 5430 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:55.685668 5437 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:55.806519 2477 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:55.807233 2368 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:55.811005 2484 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:55.817848 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:55.825145 2484 raft_consensus.cc:1261] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260806 03:51:55.821403 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 116525us
I20260806 03:51:55.691499 5095 consensus_queue.cc:1035] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.115s
W20260806 03:51:55.792146 5425 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.685s user 0.009s sys 0.005s
I20260806 03:51:55.807014 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:55.834105 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:55.838663 2436 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:55.848313 5127 raft_consensus.cc:686] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:55.805815 5385 consensus_queue.cc:260] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:55.692871 3587 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:51:55.859774 5544 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.011s
I20260806 03:51:55.865773 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:55.806778 5465 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:55.817119 5268 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:55.767652 5387 consensus_queue.cc:1046] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.088s
W20260806 03:51:55.839471 5425 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s user 0.010s sys 0.006s
I20260806 03:51:55.817884 5434 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:55.838258 5393 raft_consensus.cc:3032] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:55.875480 5504 raft_consensus.cc:2741] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260806 03:51:55.878225 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:55.893611 2576 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:55.894922 2537 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:55.853617 5437 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:55.880393 5465 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:51:55.870831 5398 raft_consensus.cc:2741] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:51:55.759259 5542 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.121s
I20260806 03:51:55.899322 3587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 361) took 1280 ms. Trace:
I20260806 03:51:55.897303 5393 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:55.936787 2474 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
W20260806 03:51:55.988541 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 273980us
I20260806 03:51:55.920718 5435 raft_consensus.cc:686] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:56.058007 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:55.970389 5320 consensus_meta.cc:287] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.874s user 0.015s sys 0.000s
I20260806 03:51:55.818789 5236 consensus_queue.cc:260] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:55.932031 5387 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:56.092610 5430 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.090915 2372 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:56.032102 5540 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:55.907228 5268 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:55.990392 5520 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 03:51:56.107602 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 212846us
I20260806 03:51:55.909534 5434 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.065255 5398 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:56.125769 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 198442us
I20260806 03:51:55.913560 5437 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:55.930325 3587 rpcz_store.cc:276] 0806 03:51:54.592842 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.593048 (+ 206us) service_pool.cc:227] Handling call
0806 03:51:55.873439 (+1280391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:51:56.141634 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:56.152069 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:56.152900 2576 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:56.155926 2538 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:56.172966 2583 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:56.198705 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:56.199582 2574 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:56.128460 5236 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:56.133080 4993 consensus_queue.cc:1046] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
W20260806 03:51:56.160321 5474 consensus_meta.cc:287] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.160s user 0.013s sys 0.005s
W20260806 03:51:56.203568 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 47791.9us
I20260806 03:51:56.204450 2372 raft_consensus.cc:1261] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:51:56.219106 2538 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:56.205221 2368 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:51:56.206997 2577 raft_consensus.cc:1261] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 4 index: 6. (term mismatch)
I20260806 03:51:56.220574 2577 pending_rounds.cc:76] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 5
I20260806 03:51:56.220888 2577 pending_rounds.cc:97] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260806 03:51:56.221182 2577 raft_consensus.cc:2866] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:56.221253 2538 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:56.137712 5518 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:56.200914 2572 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260806 03:51:56.213989 5547 log.cc:924] Time spent T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.101s user 0.000s sys 0.001s
W20260806 03:51:56.210314 5534 log.cc:924] Time spent T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.434s user 0.002s sys 0.000s
I20260806 03:51:56.207656 5544 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 03:51:56.242333 2332 leader_election.cc:334] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.602s (SENT)
I20260806 03:51:56.106230 5320 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:56.099720 5127 consensus_queue.cc:237] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:56.257400 2332 leader_election.cc:334] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.842s (SENT)
I20260806 03:51:56.258363 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:51:56.262221 3593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 364) took 1056 ms. Trace:
I20260806 03:51:56.262946 3593 rpcz_store.cc:276] 0806 03:51:55.206131 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.206311 (+ 180us) service_pool.cc:227] Handling call
0806 03:51:55.221500 (+ 15189us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:55.240793 (+ 19293us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:55.240854 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:55.241793 (+ 939us) log.cc:842] Serialized 26 byte log entry
0806 03:51:55.397960 (+156167us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:51:55.398008 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:55.398086 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:56.261981 (+863895us) raft_consensus.cc:1671] finished
0806 03:51:56.262013 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:56.262146 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16987,"raft.queue_time_us":29431,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":17450,"thread_start_us":133094,"threads_started":1,"wal-append.queue_time_us":362373}
W20260806 03:51:56.234339 5405 consensus_meta.cc:287] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.174s user 0.023s sys 0.000s
I20260806 03:51:56.274511 5095 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:51:56.227265 5329 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:56.268954 2332 leader_election.cc:334] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.772s (SENT)
I20260806 03:51:56.153044 5398 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:56.128756 5403 consensus_queue.cc:1159] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fe3d17f4ad984b9088a268d58d87469c" responder_term: 6 status { last_received { term: 5 index: 3 } last_committed_idx: 2 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:51:56.272902 5407 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:51:56.273288 5245 raft_consensus.cc:3032] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:56.128211 5393 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:56.282168 2373 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:51:56.324708 5509 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 1.239s user 0.003s sys 0.000s
I20260806 03:51:56.282058 2332 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:56.263947 5548 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.192s user 0.000s sys 0.000s
I20260806 03:51:56.249572 5518 raft_consensus.cc:686] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:56.237171 5474 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.301887 5407 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:56.302105 5545 log.cc:924] Time spent T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260806 03:51:56.304631 5320 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:56.277678 5544 raft_consensus.cc:686] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:51:56.310070 5398 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.242919 5435 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.316359 5268 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:56.254797 5511 consensus_queue.cc:1035] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260806 03:51:56.338272 5329 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:56.440096 5329 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.441112 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 174) took 1928 ms. Trace:
I20260806 03:51:56.442994 5329 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.441708 2581 rpcz_store.cc:276] 0806 03:51:54.512874 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.513102 (+ 228us) service_pool.cc:227] Handling call
0806 03:51:54.524284 (+ 11182us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:54.524807 (+ 523us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:54.524851 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:55.374790 (+849939us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b2c001765a8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:51:55.375273 (+ 483us) log.cc:842] Serialized 26 byte log entry
0806 03:51:55.738112 (+362839us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:55.738170 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:55.738277 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:56.440608 (+702331us) raft_consensus.cc:1671] finished
0806 03:51:56.440652 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:56.440987 (+ 335us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":131,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":5343,"spinlock_wait_cycles":2640640,"thread_start_us":339616,"threads_started":1,"wal-append.queue_time_us":502222}
I20260806 03:51:56.275393 5095 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:51:56.444074 5095 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:56.444592 5095 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:51:56.444531 5518 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:56.332173 5256 consensus_meta.cc:287] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.107s user 0.013s sys 0.004s
I20260806 03:51:56.348489 5560 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:51:56.358294 5561 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260806 03:51:56.301390 5245 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:56.346941 2576 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:51:56.374856 2574 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:51:56.394546 5320 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:56.281965 5405 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.410257 2373 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:51:56.419600 5398 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:56.434523 5544 consensus_queue.cc:237] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.449154 3592 raft_consensus.cc:2354] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:56.450240 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1065) took 1025 ms. Trace:
I20260806 03:51:56.374920 5434 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.497344 2369 rpcz_store.cc:276] 0806 03:51:55.425098 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.425263 (+ 165us) service_pool.cc:227] Handling call
0806 03:51:55.425667 (+ 404us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:51:55.426096 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:55.426125 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:51:55.426684 (+ 559us) log.cc:842] Serialized 52 byte log entry
0806 03:51:55.876206 (+449522us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:55.877261 (+ 1055us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:55.877702 (+ 441us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:56.439321 (+561619us) raft_consensus.cc:1671] finished
0806 03:51:56.439358 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:56.450148 (+ 10790us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":817,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":44296,"thread_start_us":376801,"threads_started":1,"wal-append.queue_time_us":692552}
I20260806 03:51:56.550988 5562 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:56.552138 5562 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:56.553624 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2103) took 1174 ms. Trace:
I20260806 03:51:56.554190 2579 rpcz_store.cc:276] 0806 03:51:55.379331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.379495 (+ 164us) service_pool.cc:227] Handling call
0806 03:51:55.409140 (+ 29645us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:55.433661 (+ 24521us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:56.553193 (+1119532us) spinlock_profiling.cc:243] Waited 9.39 ms on lock 0x7b540024d1d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:56.553527 (+ 334us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22787584}
I20260806 03:51:56.465315 5256 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:56.556378 5510 consensus_meta.cc:287] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.135s user 0.011s sys 0.004s
I20260806 03:51:56.535048 5320 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:56.559832 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1064) took 1135 ms. Trace:
I20260806 03:51:56.486572 5474 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:56.568531 5501 log.cc:924] Time spent T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 1.553s user 0.003s sys 0.000s
I20260806 03:51:56.577046 5245 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.504256 5437 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.592728 5387 raft_consensus.cc:3032] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:51:56.701711 5387 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:56.597492 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1767) took 2246 ms. Trace:
I20260806 03:51:56.704237 2482 rpcz_store.cc:276] 0806 03:51:54.351287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.353433 (+ 2146us) service_pool.cc:227] Handling call
0806 03:51:54.353897 (+ 464us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:54.557017 (+203120us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:56.597103 (+2040086us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540019fd54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:56.597345 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":464834944}
I20260806 03:51:56.703658 5387 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.715917 5256 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:56.516343 2369 pending_rounds.cc:76] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 4
W20260806 03:51:56.633268 2537 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 2.051s (SENT) [suppressed 31 similar messages]
I20260806 03:51:56.717535 2369 pending_rounds.cc:97] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260806 03:51:56.717926 2369 raft_consensus.cc:2866] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 03:51:56.718003 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 84796.5us
I20260806 03:51:56.629604 2375 rpcz_store.cc:276] 0806 03:51:55.424713 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.427166 (+ 2453us) service_pool.cc:227] Handling call
0806 03:51:55.646123 (+218957us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:51:55.677899 (+ 31776us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:55.677934 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:51:55.715999 (+ 38065us) log.cc:842] Serialized 52 byte log entry
0806 03:51:55.918664 (+202665us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:51:55.918714 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:55.918799 (+ 85us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:56.559504 (+640705us) raft_consensus.cc:1671] finished
0806 03:51:56.559547 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:56.559749 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31216,"raft.queue_time_us":32407,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":5204,"thread_start_us":119119,"threads_started":1,"wal-append.queue_time_us":469128}
I20260806 03:51:56.645563 5414 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:56.647485 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1782) took 2235 ms. Trace:
I20260806 03:51:56.513185 5398 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:56.655283 5550 log.cc:924] Time spent T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.529s user 0.004s sys 0.000s
I20260806 03:51:56.657545 5385 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:56.657986 5320 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.661526 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 191) took 1392 ms. Trace:
I20260806 03:51:56.676657 2584 raft_consensus.cc:2379] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:56.631757 5510 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:56.686600 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1797) took 2269 ms. Trace:
I20260806 03:51:56.765071 2376 rpcz_store.cc:276] 0806 03:51:55.268976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.269126 (+ 150us) service_pool.cc:227] Handling call
0806 03:51:55.288428 (+ 19302us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:55.305950 (+ 17522us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:56.661173 (+1355223us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b54000127d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:56.661452 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37681408}
I20260806 03:51:56.770350 2480 rpcz_store.cc:276] 0806 03:51:54.417289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.808361 (+391072us) service_pool.cc:227] Handling call
0806 03:51:55.665671 (+857310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:55.666112 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:55.666151 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:55.666210 (+ 59us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:55.666252 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:55.666326 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:56.686303 (+1019977us) spinlock_profiling.cc:243] Waited 857 ms on lock 0x7b54002c1850. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:56.686504 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":751,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":177,"spinlock_wait_cycles":1970918784}
I20260806 03:51:56.771979 2578 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:51:56.733299 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1809) took 2308 ms. Trace:
I20260806 03:51:56.772827 2379 rpcz_store.cc:276] 0806 03:51:54.425125 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.717679 (+292554us) service_pool.cc:227] Handling call
0806 03:51:54.776742 (+ 59063us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:55.011796 (+235054us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:55.011840 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:55.011891 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:55.011933 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:56.732888 (+1720955us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54001c0a54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:56.732970 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:56.733206 (+ 236us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":539995008}
I20260806 03:51:56.732795 5414 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:56.721261 2476 rpcz_store.cc:276] 0806 03:51:54.411919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.765816 (+353897us) service_pool.cc:227] Handling call
0806 03:51:54.766235 (+ 419us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:54.766505 (+ 270us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:54.770244 (+ 3739us) log.cc:842] Serialized 14 byte log entry
0806 03:51:56.560654 (+1790410us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:56.646457 (+ 85803us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b18002e2a60. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d7c75482c 00007f0d79b7f9dc 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9ce22 00007f0d7c7465bb 00007f0d7c74c063 00007f0d7c74c57b
0806 03:51:56.646903 (+ 446us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:56.646985 (+ 82us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:56.647054 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:56.647177 (+ 123us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:56.647398 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3742,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":64,"raft.queue_time_us":1399,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":143,"spinlock_wait_cycles":81935488,"thread_start_us":127832,"threads_started":1,"wal-append.queue_time_us":179036}
W20260806 03:51:56.753651 5236 consensus_meta.cc:287] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.542s user 0.012s sys 0.000s
W20260806 03:51:56.776412 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.719s user 0.002s sys 0.028s
W20260806 03:51:56.776850 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.749s user 0.003s sys 0.028s
W20260806 03:51:56.777184 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 787783us
I20260806 03:51:56.827742 5320 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.827855 5414 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:56.808961 5476 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:56.834831 2433 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.666s user 0.031s sys 0.074s
W20260806 03:51:56.835579 2433 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.731s user 0.031s sys 0.074s
I20260806 03:51:56.838460 5398 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.841288 5396 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:56.841941 5396 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:56.842876 5405 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.825341 5510 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:56.819818 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:56.843864 5405 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:56.859560 5405 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:56.860466 5465 consensus_meta.cc:287] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.803s user 0.008s sys 0.008s
I20260806 03:51:56.861143 5465 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.863349 5405 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:56.851440 5320 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:56.869345 5320 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.839175 5476 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:56.881518 5236 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:56.888954 5236 raft_consensus.cc:686] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:56.890614 5236 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:56.866041 5510 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:56.874109 5469 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.058s user 0.000s sys 0.017s
W20260806 03:51:56.899595 5469 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.083s user 0.000s sys 0.018s
W20260806 03:51:56.876648 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 56999.4us
I20260806 03:51:56.915482 2331 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:56.873121 5405 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:56.916484 5414 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:56.920979 5414 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:56.921041 5405 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:56.923923 2370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1054) took 2427 ms (client timeout 1524 ms). Trace:
W20260806 03:51:56.924651 2370 rpcz_store.cc:269] 0806 03:51:54.496384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:54.562814 (+ 66430us) service_pool.cc:227] Handling call
0806 03:51:56.923615 (+2360801us) spinlock_profiling.cc:243] Waited 627 ms on lock 0x7b54001eaad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:56.923814 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1441730304}
W20260806 03:51:56.940487 3592 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 354) took 1677 ms (client timeout 2001 ms). Trace:
W20260806 03:51:56.941332 3592 rpcz_store.cc:269] 0806 03:51:55.263290 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.263670 (+ 380us) service_pool.cc:227] Handling call
0806 03:51:56.926546 (+1662876us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b5400580554. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:56.940365 (+ 13819us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2589965184}
I20260806 03:51:56.942974 5437 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:56.942060 5414 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:56.945076 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2093) took 1663 ms. Trace:
I20260806 03:51:56.897806 5396 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:56.945573 2578 rpcz_store.cc:276] 0806 03:51:55.281834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.282344 (+ 510us) service_pool.cc:227] Handling call
0806 03:51:55.319638 (+ 37294us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:56.772543 (+1452905us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:56.944694 (+172151us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b540013abd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:56.944994 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3339386112}
I20260806 03:51:56.943699 5437 raft_consensus.cc:686] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
W20260806 03:51:56.947209 5095 consensus_meta.cc:287] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.502s user 0.017s sys 0.000s
I20260806 03:51:56.958189 5095 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:56.958909 5437 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:56.975183 2584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2099) took 1650 ms (client timeout 1973 ms). Trace:
I20260806 03:51:56.947960 2331 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:56.910287 5512 raft_consensus.cc:3032] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:51:56.956663 5465 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:56.975716 2584 rpcz_store.cc:269] 0806 03:51:55.324533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.324683 (+ 150us) service_pool.cc:227] Handling call
0806 03:51:56.974895 (+1650212us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b540004b854. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:56.975092 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3088692096}
I20260806 03:51:57.009980 5095 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:56.997962 5476 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:56.997337 5512 raft_consensus.cc:729] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:57.005973 5510 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:57.039377 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 91533.7us
I20260806 03:51:57.029129 5405 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:57.048532 5465 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:57.048764 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:57.049324 5465 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:57.049611 3592 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:57.049835 5465 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:57.053442 5562 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:57.059440 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:57.067026 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:57.303762 3542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.123s user 0.020s sys 0.025s
I20260806 03:51:57.418540 5387 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:57.419301 3542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.239s user 0.024s sys 0.029s
I20260806 03:51:57.087533 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:57.163067 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:57.420509 5393 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:57.323863 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.477s user 0.017s sys 0.028s
W20260806 03:51:57.421416 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.575s user 0.017s sys 0.032s
I20260806 03:51:57.421654 5426 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:57.086647 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:57.422358 5426 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:57.258814 5320 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:57.195487 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:57.259294 5430 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:57.425383 3587 raft_consensus.cc:2396] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:51:57.425153 5430 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:57.426676 5426 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:57.087038 2481 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.973s user 0.013s sys 0.028s
W20260806 03:51:57.427714 2481 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.314s user 0.014s sys 0.044s
W20260806 03:51:57.257941 2374 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.173s user 0.059s sys 0.104s
W20260806 03:51:57.428334 2374 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.369s user 0.062s sys 0.111s
I20260806 03:51:57.167387 2579 raft_consensus.cc:1226] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
W20260806 03:51:57.315972 2484 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.227s user 0.011s sys 0.021s
W20260806 03:51:57.431463 2484 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.342s user 0.011s sys 0.029s
I20260806 03:51:57.075083 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:57.338860 2331 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:57.168675 5540 consensus_meta.cc:287] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.028s user 0.015s sys 0.004s
I20260806 03:51:57.423663 2578 raft_consensus.cc:2354] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
W20260806 03:51:57.453181 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 1.61455e+06us
I20260806 03:51:57.453600 3542 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:51:57.465927 5426 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:57.301793 5407 consensus_meta.cc:287] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.910s user 0.018s sys 0.000s
I20260806 03:51:57.500448 5426 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:57.500840 5407 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:57.482116 5430 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:51:57.504449 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 51004.4us
I20260806 03:51:57.519620 5543 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:57.522789 5543 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:51:57.523591 5543 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:57.522979 5469 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:51:57.272395 5569 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:57.355703 4993 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:51:57.531111 5560 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.034s user 0.000s sys 0.019s
I20260806 03:51:57.533005 5245 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:51:57.534228 5385 consensus_meta.cc:287] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.772s user 0.013s sys 0.002s
I20260806 03:51:57.535466 5407 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:57.535962 5551 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.052s user 0.046s sys 0.095s
I20260806 03:51:57.236869 5474 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:57.432793 5575 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:51:57.588543 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 249804us
I20260806 03:51:57.592798 2331 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:57.520697 5540 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:51:57.580842 3545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.421s user 0.015s sys 0.055s
W20260806 03:51:57.594395 3545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.435s user 0.015s sys 0.055s
W20260806 03:51:57.595259 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 1.45365e+06us
I20260806 03:51:57.589079 5032 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:57.606158 2477 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.382s user 0.011s sys 0.028s
W20260806 03:51:57.608376 2477 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.384s user 0.012s sys 0.028s
W20260806 03:51:57.609782 2479 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.702s user 0.015s sys 0.020s
W20260806 03:51:57.610229 2479 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.702s user 0.016s sys 0.020s
I20260806 03:51:57.580461 5560 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:51:57.614236 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.367s user 0.004s sys 0.037s
W20260806 03:51:57.624893 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.377s user 0.004s sys 0.037s
W20260806 03:51:57.625900 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 1.40466e+06us
I20260806 03:51:57.626834 2538 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:51:57.599347 5551 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.466s user 0.046s sys 0.097s
I20260806 03:51:57.636366 2584 raft_consensus.cc:1226] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:51:57.609004 5403 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:57.647019 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:51:57.607127 5426 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:57.610572 5407 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:57.580633 4993 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:57.663710 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:57.633528 5528 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.394s user 0.018s sys 0.030s
I20260806 03:51:57.647403 5439 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:51:57.607479 5575 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:57.602759 5474 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:57.419006 5512 consensus_queue.cc:260] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:57.605062 5032 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:57.579939 5569 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:57.700766 3545 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:57.705078 3592 raft_consensus.cc:1226] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
W20260806 03:51:57.695493 5528 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.456s user 0.018s sys 0.031s
W20260806 03:51:57.702968 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 76277.5us
I20260806 03:51:57.697427 5407 raft_consensus.cc:686] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:57.720752 3593 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:51:57.725214 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.649s user 0.013s sys 0.028s
W20260806 03:51:57.725718 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.655s user 0.013s sys 0.028s
I20260806 03:51:57.659886 5403 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:57.705798 5439 raft_consensus.cc:686] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:57.703876 5320 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:57.716264 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:57.723383 2538 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:51:57.799558 2478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.656s user 0.067s sys 0.123s
W20260806 03:51:58.501998 2478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.363s user 0.142s sys 0.310s
W20260806 03:51:58.303414 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.592s user 0.051s sys 0.067s
I20260806 03:51:57.716858 5385 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:51:58.213133 3593 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
W20260806 03:51:58.173485 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 424439us
W20260806 03:51:58.503187 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.459s user 0.062s sys 0.103s
W20260806 03:51:58.343923 2429 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 2.283s user 0.001s sys 0.059s
W20260806 03:51:58.516757 2429 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 2.456s user 0.005s sys 0.059s
W20260806 03:51:58.517289 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 2.48929e+06us
I20260806 03:51:57.706302 5510 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:51:57.799481 5469 consensus_queue.cc:1035] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.765s
I20260806 03:51:58.524927 5510 raft_consensus.cc:686] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:51:58.526127 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:58.526737 2373 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:51:58.527110 5385 raft_consensus.cc:686] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
W20260806 03:51:57.799431 2372 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.470s user 0.007s sys 0.043s
W20260806 03:51:58.528362 2372 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.199s user 0.010s sys 0.076s
I20260806 03:51:58.528469 5385 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:58.297569 2377 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.034s user 0.032s sys 0.028s
W20260806 03:51:58.290840 5437 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.284s user 0.010s sys 0.043s
W20260806 03:51:57.974435 5281 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.869s user 0.034s sys 0.021s
W20260806 03:51:58.103778 5417 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.615s user 0.017s sys 0.038s
I20260806 03:51:57.982664 5032 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:51:58.522761 5595 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:58.098712 5127 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.587s user 0.019s sys 0.103s
W20260806 03:51:58.326234 2369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.600s user 0.016s sys 0.053s
I20260806 03:51:58.162060 4993 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:58.523284 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:57.857589 5505 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:58.523669 5533 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:57.726188 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:58.525933 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:57.705906 5544 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.107s user 0.007s sys 0.029s
W20260806 03:51:58.530246 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 806939us
I20260806 03:51:58.561570 2379 raft_consensus.cc:2396] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260806 03:51:58.146533 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.815s user 0.037s sys 0.098s
I20260806 03:51:57.723384 5512 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:51:58.285257 5390 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 2.010s user 0.015s sys 0.040s
W20260806 03:51:58.500001 5543 consensus_meta.cc:287] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.976s user 0.007s sys 0.007s
W20260806 03:51:58.278169 5119 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.368s user 0.041s sys 0.058s
W20260806 03:51:58.285477 5583 log.cc:924] Time spent T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.175s user 0.000s sys 0.001s
W20260806 03:51:58.143445 5465 consensus_meta.cc:287] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.093s user 0.019s sys 0.000s
I20260806 03:51:58.545415 3593 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 358) took 1476 ms. Trace:
I20260806 03:51:57.857474 5582 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:58.505630 5393 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.084s user 0.010s sys 0.007s
I20260806 03:51:58.527285 5510 consensus_queue.cc:237] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:58.410081 5575 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:51:58.549747 2369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.824s user 0.017s sys 0.058s
I20260806 03:51:58.522457 5439 consensus_queue.cc:237] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:51:58.535264 5437 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.531s user 0.014s sys 0.047s
I20260806 03:51:58.552806 2368 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:58.307116 5569 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:58.055048 5587 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:57.801028 5590 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:57.890460 5430 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:58.541296 5281 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.435s user 0.051s sys 0.028s
I20260806 03:51:58.551806 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:58.606462 2370 raft_consensus.cc:2396] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 03:51:58.569980 3586 raft_consensus.cc:2362] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5. Re-sending same reply.
I20260806 03:51:57.887266 5387 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:58.529856 2377 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.266s user 0.036s sys 0.029s
W20260806 03:51:58.562091 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.230s user 0.039s sys 0.099s
W20260806 03:51:58.620468 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 2.35138e+06us
I20260806 03:51:58.556146 5595 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:58.563123 5403 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:58.569164 2538 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:51:58.555635 5032 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:51:58.638625 5032 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:58.639601 5595 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:58.589103 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:51:58.563308 5544 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.965s user 0.014s sys 0.032s
W20260806 03:51:58.568559 5390 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 2.293s user 0.015s sys 0.045s
I20260806 03:51:58.591928 3593 rpcz_store.cc:276] 0806 03:51:57.069386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:57.069691 (+ 305us) service_pool.cc:227] Handling call
0806 03:51:58.545148 (+1475457us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b5400540454. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:51:58.545316 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":689804160}
W20260806 03:51:58.654899 2332 leader_election.cc:334] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.009s (SENT)
I20260806 03:51:58.559104 4993 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:51:58.588452 5119 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.676s user 0.041s sys 0.064s
W20260806 03:51:58.598456 2375 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.800s user 0.000s sys 0.024s
W20260806 03:51:58.663950 2375 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.865s user 0.000s sys 0.025s
I20260806 03:51:58.561028 5533 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:58.544250 5417 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.055s user 0.027s sys 0.046s
I20260806 03:51:58.604702 2373 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
W20260806 03:51:58.554560 5127 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.043s user 0.027s sys 0.116s
I20260806 03:51:58.570230 5505 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:58.598863 5393 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:58.600178 5575 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:58.593469 5465 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:58.534036 5320 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:58.588275 5543 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:58.639331 5032 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:58.605067 5582 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:58.562834 5407 consensus_queue.cc:237] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:58.668423 2368 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:51:58.622879 5430 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:58.678053 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 180) took 2806 ms. Trace:
I20260806 03:51:58.626847 5387 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:58.605074 5587 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:58.625770 5590 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:58.690299 2477 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.077s user 0.014s sys 0.020s
W20260806 03:51:58.694757 2477 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.082s user 0.015s sys 0.020s
W20260806 03:51:58.690698 2374 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.090s user 0.013s sys 0.021s
W20260806 03:51:58.682267 5435 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.244s user 0.014s sys 0.027s
I20260806 03:51:58.680269 5127 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election won for term 7
W20260806 03:51:58.695845 2374 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.104s user 0.013s sys 0.021s
I20260806 03:51:58.689266 2484 rpcz_store.cc:276] 0806 03:51:55.872047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.872220 (+ 173us) service_pool.cc:227] Handling call
0806 03:51:55.872757 (+ 537us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:51:55.903021 (+ 30264us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:55.903053 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:51:55.912207 (+ 9154us) log.cc:842] Serialized 52 byte log entry
0806 03:51:57.079945 (+1167738us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b480024c258. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d8196c7f1 00007f0d819638e7 00007f0d8196377a 00007f0d81a5795f 00007f0d81a57627 00007f0d81a6bb2c 00007f0d81a6bad8 00007f0d81a6ba58 00007f0d81a6ba13 00007f0d81a6acbc 00007f0d809ecbe6
0806 03:51:57.431970 (+352025us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:57.432054 (+ 84us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:57.432177 (+ 123us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:58.663308 (+1231131us) raft_consensus.cc:1671] finished
0806 03:51:58.663339 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:58.677940 (+ 14601us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":29730,"raft.queue_time_us":30559,"raft.run_cpu_time_us":3465,"raft.run_wall_time_us":1254401,"spinlock_wait_cycles":10581760,"thread_start_us":1342465,"threads_started":1,"wal-append.queue_time_us":2063790}
I20260806 03:51:58.698341 2376 raft_consensus.cc:1261] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:58.696802 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1822) took 2983 ms. Trace:
I20260806 03:51:58.664490 4993 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:58.697769 5435 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.260s user 0.014s sys 0.028s
W20260806 03:51:58.701370 3625 heartbeater.cc:644] Failed to heartbeat to 127.0.183.62:40035 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.0.183.62:40035 timed out after 15.000s (SENT)
I20260806 03:51:58.700155 2374 rpcz_store.cc:276] 0806 03:51:55.713013 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.713196 (+ 183us) service_pool.cc:227] Handling call
0806 03:51:55.756701 (+ 43505us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:55.757042 (+ 341us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:51:56.036572 (+279530us) log.cc:842] Serialized 14 byte log entry
0806 03:51:57.568159 (+1531587us) log.cc:842] Serialized 14 byte log entry
0806 03:51:58.696396 (+1128237us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:58.696455 (+ 59us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:58.696506 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:58.696591 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:58.696740 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":176159,"prepare.queue_time_us":634252,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":67,"raft.queue_time_us":191435,"raft.run_cpu_time_us":468,"raft.run_wall_time_us":26023,"thread_start_us":2688916,"threads_started":3,"wal-append.queue_time_us":1937038}
I20260806 03:51:58.680572 5417 consensus_queue.cc:1035] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20260806 03:51:58.680764 5393 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:58.715056 2481 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.276s user 0.020s sys 0.041s
W20260806 03:51:58.718379 2481 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.290s user 0.020s sys 0.042s
W20260806 03:51:58.708452 5518 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.675s user 0.037s sys 0.078s
I20260806 03:51:58.685690 5575 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:58.712667 5127 raft_consensus.cc:686] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:51:58.726755 5518 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.694s user 0.037s sys 0.078s
I20260806 03:51:58.740837 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1807) took 3051 ms. Trace:
W20260806 03:51:58.736114 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.134s user 0.015s sys 0.034s
W20260806 03:51:58.750073 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.156s user 0.016s sys 0.035s
W20260806 03:51:58.750437 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 1.15771e+06us
I20260806 03:51:58.748960 2477 rpcz_store.cc:276] 0806 03:51:55.689907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.821847 (+131940us) service_pool.cc:227] Handling call
0806 03:51:55.850700 (+ 28853us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:56.202987 (+352287us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:51:57.612643 (+1409656us) log.cc:842] Serialized 14 byte log entry
0806 03:51:58.695259 (+1082616us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:58.695327 (+ 68us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:58.695403 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:58.740553 (+ 45150us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54001dd3d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:58.740608 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:58.740763 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":2467890,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":64,"spinlock_wait_cycles":707241216,"thread_start_us":2466714,"threads_started":2}
I20260806 03:51:58.761276 5544 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:58.812604 2331 leader_election.cc:398] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
I20260806 03:51:58.813042 2331 leader_election.cc:401] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260806 03:51:58.810437 5245 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.225s user 0.019s sys 0.000s
I20260806 03:51:58.824996 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:58.835194 2368 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:58.829385 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:58.848502 2331 leader_election.cc:398] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:58.849001 2331 leader_election.cc:401] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260806 03:51:58.808601 5533 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:58.852813 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:58.854467 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1836) took 1092 ms. Trace:
I20260806 03:51:58.860221 2373 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:58.860383 2376 rpcz_store.cc:276] 0806 03:51:57.762039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:57.762227 (+ 188us) service_pool.cc:227] Handling call
0806 03:51:57.797923 (+ 35696us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:58.698977 (+901054us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:58.843886 (+144909us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b54001eb4d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:58.854380 (+ 10494us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2056576768}
W20260806 03:51:58.855926 2578 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.152s user 0.033s sys 0.033s
I20260806 03:51:58.861614 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:51:58.861847 2578 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.158s user 0.033s sys 0.034s
I20260806 03:51:58.859373 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:58.867594 2379 raft_consensus.cc:2379] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:58.817613 5569 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
I20260806 03:51:58.857764 2331 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:58.869628 2373 raft_consensus.cc:2379] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 6: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:58.807016 5403 consensus_queue.cc:1035] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.051s
I20260806 03:51:58.807590 5437 consensus_queue.cc:1046] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.048s
I20260806 03:51:58.801407 5544 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:58.892213 2479 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.244s user 0.019s sys 0.049s
W20260806 03:51:58.892649 2479 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.244s user 0.019s sys 0.049s
I20260806 03:51:58.854508 5587 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:58.882470 5032 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:58.873785 5582 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:58.946012 2368 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:58.959336 5393 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:58.959833 5430 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:58.966109 5544 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:58.884111 5465 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:58.957309 5387 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:58.982767 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 202) took 3052 ms. Trace:
I20260806 03:51:58.887079 5590 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:58.891404 5543 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:58.843138 5127 consensus_queue.cc:237] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:58.999871 2369 rpcz_store.cc:276] 0806 03:51:55.930300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:56.515484 (+585184us) service_pool.cc:227] Handling call
0806 03:51:56.516057 (+ 573us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:56.718878 (+202821us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:56.718919 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:56.719456 (+ 537us) log.cc:842] Serialized 26 byte log entry
0806 03:51:58.596305 (+1876849us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:58.596375 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:58.596483 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:58.982483 (+386000us) raft_consensus.cc:1671] finished
0806 03:51:58.982508 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:58.982679 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":1870412,"threads_started":1,"wal-append.queue_time_us":2261010}
I20260806 03:51:58.958022 5587 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:58.986737 5393 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:51:58.988245 5474 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:51:59.027586 5575 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:59.026647 5465 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:51:58.972365 5569 consensus_queue.cc:1046] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260806 03:51:59.032749 5465 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:59.025627 5587 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:59.024946 5236 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.542s user 0.061s sys 0.113s
W20260806 03:51:59.060913 2438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.005s sys 0.064s
W20260806 03:51:59.065876 2438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.005s user 0.006s sys 0.065s
I20260806 03:51:59.040180 5607 raft_consensus.cc:3032] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:51:59.059914 5465 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:51:59.065654 5236 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.583s user 0.061s sys 0.114s
I20260806 03:51:59.106674 2370 raft_consensus.cc:1226] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
W20260806 03:51:59.104887 5581 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.017s sys 0.013s
W20260806 03:51:59.108958 5581 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.675s user 0.017s sys 0.013s
W20260806 03:51:59.103657 5601 log.cc:924] Time spent T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.134s user 0.001s sys 0.001s
I20260806 03:51:59.124390 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1830) took 3379 ms. Trace:
I20260806 03:51:59.124972 2372 rpcz_store.cc:276] 0806 03:51:55.745392 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:56.219300 (+473908us) service_pool.cc:227] Handling call
0806 03:51:56.249190 (+ 29890us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:56.249536 (+ 346us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:56.249574 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:56.267124 (+ 17550us) log.cc:842] Serialized 26 byte log entry
0806 03:51:58.529063 (+2261939us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:58.529121 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:58.529248 (+ 127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:59.124009 (+594761us) raft_consensus.cc:1671] finished
0806 03:51:59.124056 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:59.124283 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":2199141,"threads_started":1,"wal-append.queue_time_us":2648220}
I20260806 03:51:59.092628 5607 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:59.095619 5575 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:59.149134 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 488103us
W20260806 03:51:59.158062 2377 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.000s sys 0.009s
W20260806 03:51:59.158598 2377 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.534s user 0.000s sys 0.010s
W20260806 03:51:59.104835 5580 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s user 0.012s sys 0.016s
W20260806 03:51:59.155836 2332 connection.cc:232] RPC call timeout handler was delayed by -2.1866829949431121s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2197s for a response
W20260806 03:51:59.174082 2332 leader_election.cc:334] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.884s (SENT)
I20260806 03:51:59.174566 2332 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:51:59.176116 2537 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.307s user 0.021s sys 0.072s
W20260806 03:51:59.164988 5580 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.913s user 0.014s sys 0.019s
W20260806 03:51:59.175478 5540 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.785s user 0.016s sys 0.043s
I20260806 03:51:59.180452 4993 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:59.176759 2537 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.308s user 0.022s sys 0.072s
W20260806 03:51:59.182538 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 1.31397e+06us
W20260806 03:51:59.177819 2375 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.014s sys 0.037s
W20260806 03:51:59.190541 2375 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s user 0.014s sys 0.037s
W20260806 03:51:59.180703 5540 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.790s user 0.016s sys 0.043s
I20260806 03:51:59.193306 5236 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:59.194303 5568 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:51:59.191327 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.006s sys 0.005s
W20260806 03:51:59.198072 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.661s user 0.007s sys 0.005s
I20260806 03:51:59.199517 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1837) took 1436 ms. Trace:
I20260806 03:51:59.206326 2375 rpcz_store.cc:276] 0806 03:51:57.762958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:57.763141 (+ 183us) service_pool.cc:227] Handling call
0806 03:51:57.798337 (+ 35196us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:57.798546 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:51:58.664498 (+865952us) log.cc:842] Serialized 14 byte log entry
0806 03:51:59.199088 (+534590us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:59.199153 (+ 65us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:59.199218 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:59.199307 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:59.199444 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1207973,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":49,"raft.queue_time_us":3190,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":105,"thread_start_us":1396116,"threads_started":2}
I20260806 03:51:59.210712 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:59.177187 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.642s user 0.010s sys 0.001s
W20260806 03:51:59.173060 5606 log.cc:924] Time spent T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20260806 03:51:59.158438 5575 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:59.310238 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.409922 2584 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
W20260806 03:51:59.178576 5390 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.004s sys 0.005s
I20260806 03:51:59.413113 2581 raft_consensus.cc:1226] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 6. Ops: []
W20260806 03:51:59.400701 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 142754us
I20260806 03:51:59.310238 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.414264 2572 raft_consensus.cc:2379] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:59.279098 2576 raft_consensus.cc:1261] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260806 03:51:59.396543 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 222465us
I20260806 03:51:59.194435 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 200) took 3281 ms. Trace:
W20260806 03:51:59.415683 2332 leader_election.cc:334] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.104s (SENT)
W20260806 03:51:59.283104 5385 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.020s sys 0.013s
W20260806 03:51:59.402573 2481 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.649s user 0.014s sys 0.028s
W20260806 03:51:59.419734 2481 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.014s sys 0.028s
I20260806 03:51:59.169966 5643 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:59.405216 2479 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.001s sys 0.008s
W20260806 03:51:59.422559 2479 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.529s user 0.002s sys 0.009s
I20260806 03:51:59.263952 5435 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:59.330245 5587 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:59.229079 5320 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:51:59.273068 3547 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.011s sys 0.003s
W20260806 03:51:59.429267 3547 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.740s user 0.013s sys 0.006s
I20260806 03:51:59.429522 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.416213 2332 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:51:59.351575 5629 log.cc:924] Time spent T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
W20260806 03:51:59.334096 5512 consensus_meta.cc:287] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.757s user 0.011s sys 0.006s
W20260806 03:51:59.327373 5638 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
W20260806 03:51:59.257071 5627 log.cc:924] Time spent T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260806 03:51:59.415362 2377 rpcz_store.cc:276] 0806 03:51:55.919271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:56.263038 (+343767us) service_pool.cc:227] Handling call
0806 03:51:56.263383 (+ 345us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:51:56.263614 (+ 231us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:51:58.624014 (+2360400us) log.cc:842] Serialized 14 byte log entry
0806 03:51:59.193931 (+569917us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:51:59.194009 (+ 78us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:51:59.194084 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:59.194187 (+ 103us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:59.194349 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":94,"prepare.queue_time_us":249017,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":72,"raft.queue_time_us":32774,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"thread_start_us":2890435,"threads_started":2}
W20260806 03:51:59.400701 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.870s user 0.011s sys 0.002s
I20260806 03:51:59.417519 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:59.417917 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:59.442188 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 936105us
W20260806 03:51:59.410619 5510 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.643s user 0.003s sys 0.007s
I20260806 03:51:59.420178 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:51:59.242535 5620 log.cc:924] Time spent T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.063s user 0.000s sys 0.000s
I20260806 03:51:59.311659 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:59.229542 2478 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.022s sys 0.033s
I20260806 03:51:59.423071 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1804) took 3735 ms. Trace:
W20260806 03:51:59.446555 2478 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.899s user 0.024s sys 0.033s
W20260806 03:51:59.242535 5610 log.cc:924] Time spent T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260806 03:51:59.423666 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.447443 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.448351 2584 raft_consensus.cc:2379] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:59.178409 5607 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:51:59.416927 5390 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.740s user 0.005s sys 0.014s
I20260806 03:51:59.398892 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:51:59.298178 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.008s sys 0.007s
I20260806 03:51:59.451393 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1812) took 3520 ms. Trace:
W20260806 03:51:59.429716 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 741162us
I20260806 03:51:59.451787 2579 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
W20260806 03:51:59.243438 5622 log.cc:924] Time spent T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
W20260806 03:51:59.431159 2332 leader_election.cc:334] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 3.028s (SENT)
I20260806 03:51:59.453203 2332 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:59.409744 5281 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:51:59.419652 5385 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s user 0.022s sys 0.024s
I20260806 03:51:59.427951 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1806) took 3738 ms. Trace:
W20260806 03:51:59.469204 2332 connection.cc:232] RPC call timeout handler was delayed by -2.1362846498377621s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1502s for a response
W20260806 03:51:59.469650 2332 connection.cc:232] RPC call timeout handler was delayed by -2.1186159648932517s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260806 03:51:59.470098 2332 connection.cc:232] RPC call timeout handler was delayed by -2.0952159648295492s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2175s for a response
W20260806 03:51:59.470633 2332 connection.cc:232] RPC call timeout handler was delayed by -1.9818055159412324s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
W20260806 03:51:59.470916 2332 connection.cc:232] RPC call timeout handler was delayed by -1.9103846498765051s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1753s for a response
I20260806 03:51:59.434917 2484 raft_consensus.cc:1226] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: []
I20260806 03:51:59.437748 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:59.412766 3586 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:51:59.449041 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:59.450543 2572 raft_consensus.cc:2379] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:59.474403 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:59.475385 2480 raft_consensus.cc:2354] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:59.450313 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.476847 2477 raft_consensus.cc:2396] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260806 03:51:59.453676 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.009s sys 0.007s
W20260806 03:51:59.451749 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.814s user 0.008s sys 0.012s
I20260806 03:51:59.431619 5505 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:59.451845 2478 rpcz_store.cc:276] 0806 03:51:55.930601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:56.069729 (+139128us) service_pool.cc:227] Handling call
0806 03:51:56.097728 (+ 27999us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:51:56.098160 (+ 432us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:56.098186 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:51:56.098846 (+ 660us) log.cc:842] Serialized 52 byte log entry
0806 03:51:58.502775 (+2403929us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:51:59.447010 (+944235us) log.cc:842] Serialized 14 byte log entry
0806 03:51:59.447447 (+ 437us) log.cc:842] Serialized 14 byte log entry
0806 03:51:59.447664 (+ 217us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:59.447742 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:59.447786 (+ 44us) raft_consensus.cc:1671] finished
0806 03:51:59.447797 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:59.451325 (+ 3528us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5759,"prepare.queue_time_us":186893,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":4293,"raft.queue_time_us":63901,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":13987,"thread_start_us":3263861,"threads_started":2,"wal-append.queue_time_us":2849191}
I20260806 03:51:59.452152 2534 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:51:59.481195 2484 raft_consensus.cc:1226] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 1. Current term is 2. Ops: []
I20260806 03:51:59.434778 5568 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:59.477214 2576 raft_consensus.cc:2379] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 2: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:51:59.477569 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 908485us
W20260806 03:51:59.458168 2236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38220 (request call id 127) took 15923 ms (client timeout 15000 ms). Trace:
W20260806 03:51:59.477294 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.602s user 0.009s sys 0.008s
W20260806 03:51:59.489039 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 631245us
I20260806 03:51:59.489797 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:59.469396 2479 rpcz_store.cc:276] 0806 03:51:55.689534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.829181 (+139647us) service_pool.cc:227] Handling call
0806 03:51:55.829697 (+ 516us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:55.830191 (+ 494us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:55.830231 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:55.838316 (+ 8085us) log.cc:842] Serialized 26 byte log entry
0806 03:51:57.610833 (+1772517us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:58.893059 (+1282226us) log.cc:842] Serialized 14 byte log entry
0806 03:51:59.423155 (+530096us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:59.423312 (+ 157us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:59.423362 (+ 50us) raft_consensus.cc:1671] finished
0806 03:51:59.423374 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:59.427874 (+ 4500us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7153,"prepare.queue_time_us":1073075,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":60,"raft.queue_time_us":173539,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":33810,"thread_start_us":3477176,"threads_started":3,"wal-append.queue_time_us":2404387}
W20260806 03:51:59.479259 2332 connection.cc:232] RPC call timeout handler was delayed by -1.7022519060410559s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2096s for a response
W20260806 03:51:59.490653 2332 connection.cc:232] RPC call timeout handler was delayed by -1.651515964884311s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26680000000000004s for a response
W20260806 03:51:59.491245 2332 connection.cc:232] RPC call timeout handler was delayed by -1.5230985968373716s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1695s for a response
W20260806 03:51:59.459388 5510 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s user 0.004s sys 0.007s
I20260806 03:51:59.492648 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.497020 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:59.498420 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:59.505381 2484 raft_consensus.cc:2354] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
W20260806 03:51:59.491631 2332 connection.cc:232] RPC call timeout handler was delayed by -1.4991266739089042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20260806 03:51:59.446691 2481 rpcz_store.cc:276] 0806 03:51:55.687661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:55.804897 (+117236us) service_pool.cc:227] Handling call
0806 03:51:55.832291 (+ 27394us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:55.841826 (+ 9535us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:55.841875 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:55.842471 (+ 596us) log.cc:842] Serialized 26 byte log entry
0806 03:51:56.087003 (+244532us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:51:56.718435 (+631432us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b480024c258. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d8196c7f1 00007f0d819638e7 00007f0d8196377a 00007f0d81a5795f 00007f0d81a57627 00007f0d81a6bb2c 00007f0d81a6bad8 00007f0d81a6ba58 00007f0d81a6ba13 00007f0d81a6acbc 00007f0d809ecbe6
0806 03:51:57.428287 (+709852us) log.cc:842] Serialized 14 byte log entry
0806 03:51:59.420217 (+1991930us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:59.420301 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:59.420349 (+ 48us) raft_consensus.cc:1671] finished
0806 03:51:59.422817 (+ 2468us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:59.423011 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8976,"prepare.queue_time_us":2545082,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":331773,"raft.run_cpu_time_us":4824,"raft.run_wall_time_us":883889,"spinlock_wait_cycles":15713024,"thread_start_us":3493531,"threads_started":4,"wal-append.queue_time_us":2023738,"wal-append.run_cpu_time_us":1962,"wal-append.run_wall_time_us":1001993}
I20260806 03:51:59.501593 2476 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
W20260806 03:51:59.507570 2332 connection.cc:232] RPC call timeout handler was delayed by -1.3390820138156414s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1958s for a response
W20260806 03:51:59.508440 2332 connection.cc:232] RPC call timeout handler was delayed by -1.2231159647926688s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3144s for a response
W20260806 03:51:59.508759 2332 connection.cc:232] RPC call timeout handler was delayed by -1.1648985969368368s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2093s for a response
W20260806 03:51:59.509050 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0538892070762813s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2944s for a response
W20260806 03:51:59.509332 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0359891247935593s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2501s for a response
I20260806 03:51:59.510417 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.468416 5320 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:59.517486 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.518515 2375 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:59.519886 2584 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260806 03:51:59.520329 2584 pending_rounds.cc:76] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 2
I20260806 03:51:59.520560 2584 pending_rounds.cc:97] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 03:51:59.520803 2584 raft_consensus.cc:2866] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:51:59.485852 5512 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:59.532891 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:51:59.500241 5127 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:51:59.436199 5435 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:59.423411 5643 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:59.484822 5587 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:59.502961 5582 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:59.565052 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.547411 2581 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
W20260806 03:51:59.487841 2236 rpcz_store.cc:269] 0806 03:51:43.534979 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:43.535248 (+ 269us) service_pool.cc:227] Handling call
0806 03:51:43.839305 (+304057us) spinlock_profiling.cc:243] Waited 52.7 ms on lock 0x7b54000e3154. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4aaf0 00007f0d7cacdf2a 00007f0d82a1fd6c 00007f0d82b19c3e 00007f0d80e2bfb4 00007f0d80e2bf42 00007f0d80e2be71 00007f0d80e2bdec 00007f0d80e2b0a2 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:51:59.458071 (+15618766us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:51:53.148965 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:51:53.323072 (+174107us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:51:53.326338 (+ 3266us) write_op.cc:435] Acquired schema lock
0806 03:51:53.326367 (+ 29us) tablet.cc:660] Decoding operations
0806 03:51:53.716141 (+389774us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:51:53.734251 (+ 18110us) write_op.cc:641] Partition lock acquired for write op
0806 03:51:53.757135 (+ 22884us) tablet.cc:683] Acquiring locks for 10 operations
0806 03:51:53.882401 (+125266us) tablet.cc:699] Row locks acquired
0806 03:51:53.882428 (+ 27us) write_op.cc:260] PREPARE: finished
0806 03:51:53.882624 (+ 196us) write_op.cc:270] Start()
0806 03:51:53.882774 (+ 150us) write_op.cc:276] Timestamp: P: 1785988313882572 usec, L: 0
0806 03:51:53.882797 (+ 23us) op_driver.cc:331] REPLICATION: starting
0806 03:51:53.883640 (+ 843us) log.cc:842] Serialized 3985 byte log entry
0806 03:51:55.140123 (+1256483us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d79b7f5fe 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9ce22 00007f0d7c7465bb
0806 03:51:56.064157 (+924034us) op_driver.cc:444] REPLICATION: finished
0806 03:51:57.990458 (+1926301us) write_op.cc:301] APPLY: starting
0806 03:51:58.026228 (+ 35770us) tablet.cc:1365] starting BulkCheckPresence
0806 03:51:58.248739 (+222511us) tablet.cc:1368] finished BulkCheckPresence
0806 03:51:58.248770 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:51:58.250426 (+ 1656us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:51:58.250818 (+ 392us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0806 03:51:58.250891 (+ 73us) write_op.cc:312] APPLY: finished
0806 03:51:58.251383 (+ 492us) log.cc:842] Serialized 77 byte log entry
0806 03:51:59.234998 (+983615us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:59.264865 (+ 29867us) write_op.cc:454] Released schema lock
0806 03:51:59.284924 (+ 20059us) write_op.cc:341] FINISH: Updating metrics
0806 03:51:59.330432 (+ 45508us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:51:59.330500 (+ 68us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":10057,"reported_tablets":15,"spinlock_wait_cycles":121192320,"child_traces":[["op",{"apply.queue_time_us":1879862,"apply.run_cpu_time_us":36686,"apply.run_wall_time_us":1344101,"num_ops":10,"prepare.queue_time_us":326612,"prepare.run_cpu_time_us":71813,"prepare.run_wall_time_us":2291700,"raft.queue_time_us":521149,"raft.run_cpu_time_us":145580,"raft.run_wall_time_us":1708916,"replication_time_us":2151825,"spinlock_wait_cycles":4024704,"thread_start_us":4511669,"threads_started":5,"wal-append.queue_time_us":2349291,"wal-append.run_cpu_time_us":15479,"wal-append.run_wall_time_us":1492297}]]}
I20260806 03:51:59.549489 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.574965 2476 raft_consensus.cc:2396] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:59.555747 2577 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:51:59.556561 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:59.576092 2479 raft_consensus.cc:2354] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
I20260806 03:51:59.558056 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.576848 2481 raft_consensus.cc:2379] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:51:59.557958 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:59.561770 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:59.549055 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.581763 3587 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
W20260806 03:51:59.579332 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 46551.9us
I20260806 03:51:59.583231 2331 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:59.529031 5032 consensus_queue.cc:1035] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.055s
I20260806 03:51:59.584122 5645 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260806 03:51:59.579200 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.590951 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.580717 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.594414 2370 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:51:59.581161 3586 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEARNER]: Advancing to term 4
I20260806 03:51:59.599853 2575 raft_consensus.cc:1226] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 5. Ops: []
I20260806 03:51:59.587852 2484 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:51:59.588657 2574 raft_consensus.cc:1226] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:51:59.590133 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.601696 2377 raft_consensus.cc:1226] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 2. Current term is 3. Ops: []
I20260806 03:51:59.602033 2479 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:51:59.598687 3542 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:59.604211 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.477093 5281 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:59.579952 5660 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:51:59.614710 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:51:59.615824 2574 raft_consensus.cc:2379] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:51:59.596871 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.544605 5568 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:51:59.530517 5607 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:51:59.608417 4993 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:59.610849 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.624370 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:59.613328 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.629290 2477 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:59.619021 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.620739 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.604887 5236 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:59.632905 2484 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:51:59.642580 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.650219 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:51:59.649480 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 114766us
I20260806 03:51:59.568965 5465 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:59.551962 5512 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:59.586721 5587 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:51:59.605000 5582 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:51:59.649546 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 132039us
I20260806 03:51:59.848766 2474 raft_consensus.cc:3032] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:59.846838 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.730036 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:59.711275 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:59.677346 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.847424 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.643534 2372 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:59.600324 5320 raft_consensus.cc:2781] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:51:59.838841 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.709547 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.703719 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.861920 2474 raft_consensus.cc:729] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:51:59.762102 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.816859 2574 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: []
I20260806 03:51:59.866441 2484 raft_consensus.cc:2379] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:59.756065 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.867820 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.868654 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.871892 3590 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:59.650992 3593 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:51:59.872418 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:59.877271 5643 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:59.877174 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.907742 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.881443 2479 raft_consensus.cc:2396] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:59.612108 5625 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:51:59.887403 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.907940 3584 raft_consensus.cc:2379] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:51:59.909245 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 184) took 2308 ms. Trace:
I20260806 03:51:59.889626 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.891414 2474 consensus_queue.cc:260] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:59.892805 3585 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:51:59.893115 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.893822 5320 raft_consensus.cc:686] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:51:59.652273 4993 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:51:59.894340 3592 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 2 LEARNER]: Advancing to term 6
I20260806 03:51:59.944375 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:59.895087 5666 log.cc:924] Time spent T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
W20260806 03:51:59.895840 5644 log.cc:924] Time spent T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.185s user 0.000s sys 0.000s
I20260806 03:51:59.896091 5587 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:59.898063 5465 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:59.898820 3594 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:52:00.018250 2474 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:51:59.812642 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:51:59.902110 5582 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:51:59.903112 5435 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:51:59.903631 5540 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:51:59.904927 5639 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:59.663018 5236 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:00.022238 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.782393 3587 raft_consensus.cc:1226] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 5. Ops: []
I20260806 03:51:59.906837 5640 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:51:59.906994 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:59.813198 5650 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 03:51:59.913570 2377 raft_consensus.cc:2396] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:51:59.997225 2478 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:51:59.997921 2368 raft_consensus.cc:2396] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:51:59.625608 5505 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:51:59.734386 5387 raft_consensus.cc:3032] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:51:59.998595 2477 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:51:59.951556 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:51:59.944866 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:51:59.948561 2576 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:59.951324 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:51:59.949821 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.000571 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:51:59.949826 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:51:59.949390 2572 raft_consensus.cc:1261] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:51:59.947602 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:00.018530 5439 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:00.006824 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.035660 2369 raft_consensus.cc:2362] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5. Re-sending same reply.
I20260806 03:52:00.044976 2575 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:00.008273 5407 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:00.011255 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:51:59.894384 5655 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20260806 03:52:00.012008 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.012835 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:00.048936 2375 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:51:59.909476 5430 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:00.013687 3586 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEARNER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 03:52:00.013805 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 107723us
I20260806 03:51:59.951985 3584 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260806 03:51:59.913870 2578 rpcz_store.cc:276] 0806 03:51:57.601358 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:57.601549 (+ 191us) service_pool.cc:227] Handling call
0806 03:51:57.633202 (+ 31653us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:51:57.637927 (+ 4725us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:57.637962 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:51:57.662542 (+ 24580us) log.cc:842] Serialized 52 byte log entry
0806 03:51:58.864503 (+1201961us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:51:59.196149 (+331646us) log.cc:842] Serialized 14 byte log entry
0806 03:51:59.196594 (+ 445us) log.cc:842] Serialized 14 byte log entry
0806 03:51:59.197181 (+ 587us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:59.197269 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:51:59.350714 (+153445us) spinlock_profiling.cc:243] Waited 7.23 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:51:59.901177 (+550463us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b2400227eb8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d799f5cf4 00007f0d799f3dfb 00007f0d79a82559 00007f0d79a821e5 00007f0d79a81740 00007f0d7c774578 00007f0d7c774cd8 00007f0d7c74c6e8 00007f0d7c747aea 00007f0d7c746f36
0806 03:51:59.908992 (+ 7815us) raft_consensus.cc:1671] finished
0806 03:51:59.909013 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:51:59.909151 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":583853,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":73,"raft.queue_time_us":1218,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":18558,"spinlock_wait_cycles":42639488,"thread_start_us":1482952,"threads_started":2,"wal-append.queue_time_us":1472905}
I20260806 03:52:00.016085 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:51:59.642277 5624 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:59.644094 5660 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:51:59.711490 5575 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:00.028167 2538 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:00.030309 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.103096 2474 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:00.030471 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:51:59.996168 5474 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:00.035384 2484 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:52:00.038560 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.105086 2534 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:00.105934 2482 raft_consensus.cc:2396] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:00.106087 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.109097 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.001883 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.113636 2574 raft_consensus.cc:2379] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:00.114287 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.035002 2379 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
I20260806 03:52:00.044047 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.047080 2537 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:00.114946 2479 raft_consensus.cc:2362] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5. Re-sending same reply.
I20260806 03:52:00.077342 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.050763 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:00.115744 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 68755.5us
W20260806 03:52:00.084276 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 62657.8us
I20260806 03:52:00.077342 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.085353 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:00.079226 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.087643 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.087966 2577 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:00.120373 2483 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:00.091940 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.028007 5127 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:00.094336 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 78397.4us
I20260806 03:52:00.017807 5625 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:00.029942 5505 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:00.101987 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.033682 5387 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:00.051558 3594 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
W20260806 03:52:00.120675 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 92629.6us
I20260806 03:52:00.121721 2477 raft_consensus.cc:2396] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:52:00.099514 5437 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:00.123278 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.032387 5640 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:00.102929 5281 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.027446 5639 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:00.077666 5407 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:00.096553 5430 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:00.079753 5439 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:00.206090 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 207) took 1377 ms. Trace:
I20260806 03:52:00.213579 2374 rpcz_store.cc:276] 0806 03:51:58.828634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:58.828782 (+ 148us) service_pool.cc:227] Handling call
0806 03:51:58.857965 (+ 29183us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:51:58.875339 (+ 17374us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:51:58.875392 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:51:58.876140 (+ 748us) log.cc:842] Serialized 26 byte log entry
0806 03:51:59.193914 (+317774us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:51:59.193967 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:51:59.195850 (+ 1883us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:00.205766 (+1009916us) raft_consensus.cc:1671] finished
0806 03:52:00.205799 (+ 33us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:00.206007 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":907,"raft.queue_time_us":160712,"raft.run_cpu_time_us":398,"raft.run_wall_time_us":11890,"thread_start_us":231860,"threads_started":1,"wal-append.queue_time_us":700221}
I20260806 03:52:00.148125 5591 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:00.169895 5505 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:00.168495 5437 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:00.209992 5664 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20260806 03:52:00.204222 5435 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:00.205189 5510 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:52:00.239636 2575 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:00.197441 5587 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:00.118333 5474 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:00.262071 5654 log.cc:924] Time spent T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.693s user 0.005s sys 0.000s
I20260806 03:52:00.308091 5387 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:00.310387 5676 log.cc:924] Time spent T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.196s user 0.000s sys 0.001s
I20260806 03:52:00.235117 4993 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:00.315722 5675 log.cc:924] Time spent T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
I20260806 03:52:00.291621 5510 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:00.225762 5591 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:00.112073 5320 consensus_queue.cc:237] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:00.109321 5660 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:00.372105 5665 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.334s user 0.002s sys 0.002s
I20260806 03:52:00.258496 5437 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:00.421093 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.422106 2474 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:00.314114 5512 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.103387 5624 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:00.338129 5677 log.cc:924] Time spent T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.214s user 0.001s sys 0.000s
I20260806 03:52:00.389768 3591 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:00.394985 5510 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:00.404625 5660 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.395323 5591 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:00.454515 5676 log.cc:924] Time spent T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260806 03:52:00.357095 5587 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:00.457986 5587 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.408360 5430 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:00.410248 5639 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:00.412299 5439 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:00.415784 5407 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.416342 5236 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.419721 3542 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:00.483433 5474 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.421193 5387 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:52:00.483696 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 64096.2us
I20260806 03:52:00.484185 5474 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:00.484930 5474 raft_consensus.cc:686] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:00.485831 5474 consensus_queue.cc:237] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:00.500640 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 376185us
W20260806 03:52:00.504501 2436 service_pool.cc:137] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:54212 dropped due to backpressure. The service queue is full; it has 50 items. [suppressed 4 similar messages]
I20260806 03:52:00.510819 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.510599 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.511809 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.512756 2484 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:00.506014 3591 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:00.514560 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.515122 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.515523 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.516105 2482 raft_consensus.cc:2396] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:52:00.516664 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.515933 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.517251 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.516467 2481 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:52:00.519063 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.519271 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.519693 2480 raft_consensus.cc:2354] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
I20260806 03:52:00.519902 2481 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:52:00.520423 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:00.520882 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.521080 2379 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:00.522131 2478 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:00.519996 2474 raft_consensus.cc:2396] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:52:00.524981 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.525323 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:00.526084 2373 raft_consensus.cc:2379] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:00.526094 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:00.545178 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 440120us
I20260806 03:52:00.547118 2534 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:00.549621 2474 raft_consensus.cc:2354] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:52:00.526799 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:00.550537 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.551492 2474 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:00.552533 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.553597 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.554246 2474 raft_consensus.cc:2354] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
I20260806 03:52:00.554960 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.550475 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.555954 2484 raft_consensus.cc:2354] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
W20260806 03:52:00.556761 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 439746us
I20260806 03:52:00.557476 2538 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:00.557502 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.558533 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.559055 2484 raft_consensus.cc:2396] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:00.559547 2481 raft_consensus.cc:2396] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:00.559693 2534 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:00.560562 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 443865us
I20260806 03:52:00.566083 2538 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:00.567445 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.568197 2482 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:00.570963 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.571590 2480 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:00.572207 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.507480 5640 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:00.507355 5624 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.575281 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.577092 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.577855 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.564464 2584 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:00.507077 5625 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.585726 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.586728 2477 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:00.552955 5236 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:00.600430 5236 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:00.605180 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.597834 5581 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:00.610904 2480 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:00.616552 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1013) took 1070 ms. Trace:
I20260806 03:52:00.617112 2584 rpcz_store.cc:276] 0806 03:51:59.546629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:51:59.546827 (+ 198us) service_pool.cc:227] Handling call
0806 03:52:00.616124 (+1069297us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b540024d1d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:00.616466 (+ 342us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1292468480}
I20260806 03:52:00.554316 2436 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:54212 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 191)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2031)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 189)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2041)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2042)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2063)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2036)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1825)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2032)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2064)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1834)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2051)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1817)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1819)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1821)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1822)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1823)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1827)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1828)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1830)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1831)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1835)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1838)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1839)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1840)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1841)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1842)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1843)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1844)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1846)
W20260806 03:52:00.631184 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 458432us
I20260806 03:52:00.595785 5640 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:00.676863 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.677505 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.678362 2480 raft_consensus.cc:2354] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:52:00.679314 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.682122 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.684158 2484 raft_consensus.cc:2396] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:52:00.685134 2480 raft_consensus.cc:2396] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:52:00.595578 5685 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:00.694032 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.612087 5581 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:00.676893 5435 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:00.699860 2480 raft_consensus.cc:1261] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:00.702157 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.702900 2477 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:00.710841 2480 raft_consensus.cc:1226] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
W20260806 03:52:00.739014 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 174851us
I20260806 03:52:00.753178 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:00.697718 5607 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.054s user 0.007s sys 0.007s
I20260806 03:52:00.694101 5430 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:00.767040 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.776935 2482 raft_consensus.cc:2379] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:00.767237 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.767895 5435 raft_consensus.cc:686] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:00.778326 2476 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
W20260806 03:52:00.787783 5681 log.cc:924] Time spent T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260806 03:52:00.787732 5435 consensus_queue.cc:237] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:00.783298 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.702224 5581 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:00.789793 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.714749 5625 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:00.752734 5680 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260806 03:52:00.787932 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:00.792316 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 228481us
I20260806 03:52:00.802248 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.799623 2480 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:00.809784 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:00.812094 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 246086us
I20260806 03:52:00.815090 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.815765 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.810721 2482 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:00.816437 2480 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:00.817198 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.817610 2480 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:00.818135 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:00.793202 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 78653.3us
I20260806 03:52:00.796257 2479 raft_consensus.cc:2396] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:52:00.796759 2478 raft_consensus.cc:2379] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:00.797981 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 271167us
I20260806 03:52:00.816062 2481 raft_consensus.cc:2396] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:52:00.821101 3547 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
W20260806 03:52:00.820779 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 78407.8us
I20260806 03:52:00.791868 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.822220 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.823215 2476 raft_consensus.cc:2396] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:52:00.827013 3547 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:00.830389 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.831035 3547 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:00.832716 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.837647 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.839067 2478 raft_consensus.cc:2354] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:52:00.845288 2481 raft_consensus.cc:1261] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:00.796265 5685 raft_consensus.cc:686] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:00.848742 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.849588 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.849831 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.850478 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.856638 2374 pending_rounds.cc:76] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 5
I20260806 03:52:00.864854 2374 pending_rounds.cc:97] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260806 03:52:00.865163 2374 raft_consensus.cc:2866] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:00.865701 2429 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:00.851076 3547 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:00.857131 2479 raft_consensus.cc:2396] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:52:00.857652 2482 raft_consensus.cc:2354] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:52:00.856638 2481 pending_rounds.cc:76] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 2
I20260806 03:52:00.872165 2481 pending_rounds.cc:97] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260806 03:52:00.872462 2481 raft_consensus.cc:2866] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:00.875890 2331 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:00.853721 2474 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:00.885800 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.894573 2480 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:00.863332 5582 raft_consensus.cc:2741] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260806 03:52:00.895876 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.898002 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.855424 5119 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:00.907341 2479 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:00.871619 5663 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:00.806272 5625 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:00.855245 5684 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:52:00.883498 5439 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:00.967739 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.969441 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:00.886696 2572 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:52:00.902846 5697 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:00.914745 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.916698 5510 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:00.879580 5236 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:00.915733 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.918447 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.975354 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.830543 5540 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260806 03:52:00.990972 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.991554 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.992085 2479 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 2: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:00.907181 5119 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:00.883109 5575 consensus_queue.cc:1046] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.098s
W20260806 03:52:00.969368 5655 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20260806 03:52:00.883157 5624 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.098s
W20260806 03:52:00.904531 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 78493us
I20260806 03:52:00.967674 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:00.954335 5663 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:01.039377 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:00.972551 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 101218us
I20260806 03:52:01.043530 2480 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:52:01.043792 2332 leader_election.cc:334] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.695s (SENT)
W20260806 03:52:01.045668 2332 leader_election.cc:334] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.784s (SENT)
W20260806 03:52:01.047024 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 176436us
I20260806 03:52:00.981987 2483 raft_consensus.cc:2354] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
W20260806 03:52:00.982869 2218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 208205us
W20260806 03:52:01.047338 2332 leader_election.cc:334] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.828s (SENT)
I20260806 03:52:01.048652 2332 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:01.055125 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:00.971143 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.033742 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.968714 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:01.028529 5591 consensus_meta.cc:287] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.574s user 0.012s sys 0.000s
I20260806 03:52:01.041004 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:00.990677 5439 raft_consensus.cc:686] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:01.061296 2373 raft_consensus.cc:2379] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:01.069339 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.069541 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:01.072969 2479 raft_consensus.cc:2379] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:01.067368 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.064496 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:01.018126 5119 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:01.078922 2583 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:01.078456 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:01.162948 2373 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:52:01.084893 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:01.086138 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.076267 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:00.954399 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 138634us
I20260806 03:52:01.165071 2379 raft_consensus.cc:2396] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:01.079380 2572 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:01.166579 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.138145 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:01.046679 5663 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:00.893556 5685 consensus_queue.cc:237] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:01.169049 2370 raft_consensus.cc:2396] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:01.167908 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.185750 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.186460 2574 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:01.187652 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.076273 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:01.193753 5708 log.cc:924] Time spent T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.147s user 0.000s sys 0.000s
I20260806 03:52:01.195923 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.197227 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.197994 5624 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:01.198307 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.199443 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.167522 2477 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:01.178289 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.178599 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.047313 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:01.009442 5645 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.063s
I20260806 03:52:01.199951 5624 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:01.157673 5575 raft_consensus.cc:3032] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:52:01.234755 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.216187 2584 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:01.191668 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:01.240453 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 45202.3us
I20260806 03:52:01.218053 5439 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.220492 5510 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:01.253309 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:00.968799 5437 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.225769 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 1825) took 1343 ms. Trace:
I20260806 03:52:01.263681 2477 rpcz_store.cc:276] 0806 03:51:59.882316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:00.681224 (+798908us) service_pool.cc:227] Handling call
0806 03:52:01.225686 (+544462us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:01.263777 2537 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:01.265079 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.265981 2477 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:01.267262 2574 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260806 03:52:01.247227 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.254608 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.261744 2576 raft_consensus.cc:1261] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:52:01.268921 2574 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260806 03:52:01.269618 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 222284us
W20260806 03:52:01.270651 2332 leader_election.cc:334] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.874s (SENT)
I20260806 03:52:01.271143 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:01.277685 2370 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:52:01.181051 5320 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:01.283046 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:01.312620 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:01.314644 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.315707 2474 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:01.180106 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:01.305610 2331 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:01.079001 5591 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.253343 5689 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:01.329072 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:01.333380 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.367301 2477 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:01.244589 5575 raft_consensus.cc:729] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:01.256665 5624 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:01.367641 5032 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:01.370342 5723 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:01.360311 2480 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:01.441431 2577 raft_consensus.cc:2379] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:01.369514 2331 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:01.442885 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 172263us
I20260806 03:52:01.395134 5725 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:01.444741 5725 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:01.445393 5725 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:01.450500 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 81088.9us
W20260806 03:52:01.443845 2332 leader_election.cc:334] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.958s (SENT)
I20260806 03:52:01.400051 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1843) took 1503 ms. Trace:
I20260806 03:52:01.410925 2583 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:01.455585 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:01.457248 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.455904 2481 rpcz_store.cc:276] 0806 03:51:59.896251 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:00.856131 (+959880us) service_pool.cc:227] Handling call
0806 03:52:00.856445 (+ 314us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:00.873262 (+ 16817us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:00.873295 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:00.874109 (+ 814us) log.cc:842] Serialized 26 byte log entry
0806 03:52:00.955931 (+ 81822us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:00.955973 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:00.956084 (+ 111us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:01.399701 (+443617us) raft_consensus.cc:1671] finished
0806 03:52:01.399728 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:01.399954 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":64438,"threads_started":1,"wal-append.queue_time_us":293869}
I20260806 03:52:01.477555 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:01.488595 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 224449us
I20260806 03:52:01.325708 5533 consensus_queue.cc:1035] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.167s
I20260806 03:52:01.280730 5701 consensus_queue.cc:1046] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.169s
I20260806 03:52:01.381628 5437 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:01.374874 5624 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:01.388285 5387 consensus_meta.cc:287] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.904s user 0.008s sys 0.007s
I20260806 03:52:01.487650 5729 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:01.335439 5320 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:01.498870 5505 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.216s user 0.017s sys 0.000s
I20260806 03:52:01.373382 5625 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:01.512820 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 269803us
W20260806 03:52:01.516351 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 252666us
I20260806 03:52:01.516588 3545 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:01.503043 5723 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:01.523476 3545 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:01.526909 3545 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:01.527921 5712 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.293s user 0.000s sys 0.001s
I20260806 03:52:01.528439 5727 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:52:01.524412 5505 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.535434 5505 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:01.537024 5320 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:01.538049 3594 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:01.541895 5591 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:01.528796 5281 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:01.401350 5689 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.521245 5684 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:01.512693 5575 consensus_queue.cc:260] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:01.556806 5575 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:52:01.562588 5718 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260806 03:52:01.567847 5725 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:01.623848 2331 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:01.632408 2331 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:01.634248 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:01.475209 2332 leader_election.cc:334] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.093s (SENT)
I20260806 03:52:01.636706 2332 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:01.639256 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 164103us
W20260806 03:52:01.640690 2332 leader_election.cc:334] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.096s (SENT)
I20260806 03:52:01.641398 2332 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:01.645738 2332 leader_election.cc:334] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.149s (SENT)
W20260806 03:52:01.647933 2332 leader_election.cc:334] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.175s (SENT)
I20260806 03:52:01.651966 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:01.654947 2369 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:01.652766 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:01.656863 2370 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:01.658735 2377 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:01.659436 2370 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260806 03:52:01.659986 2370 pending_rounds.cc:76] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 2
I20260806 03:52:01.664803 2370 pending_rounds.cc:97] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 03:52:01.665284 2370 raft_consensus.cc:2866] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 03:52:01.659832 2332 leader_election.cc:334] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.197s (SENT)
W20260806 03:52:01.667946 2332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.202s (SENT) [suppressed 21 similar messages]
I20260806 03:52:01.669063 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1835) took 1782 ms. Trace:
I20260806 03:52:01.508970 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:01.662145 2429 leader_election.cc:334] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.524s (SENT)
I20260806 03:52:01.496918 2481 raft_consensus.cc:2354] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
I20260806 03:52:01.673280 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1840) took 1777 ms. Trace:
I20260806 03:52:01.674028 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.675318 2481 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:01.676856 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.676789 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.677925 2479 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:01.679787 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.669524 2476 rpcz_store.cc:276] 0806 03:51:59.886655 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:00.831920 (+945265us) service_pool.cc:227] Handling call
0806 03:52:00.868674 (+ 36754us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:00.976558 (+107884us) raft_consensus.cc:1520] Early marking committed up to index 2
0806 03:52:01.030094 (+ 53536us) log.cc:842] Serialized 14 byte log entry
0806 03:52:01.397214 (+367120us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:01.448650 (+ 51436us) log.cc:842] Serialized 26 byte log entry
0806 03:52:01.448843 (+ 193us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:01.448902 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:01.449004 (+ 102us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:01.668402 (+219398us) raft_consensus.cc:1671] finished
0806 03:52:01.668442 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:01.668799 (+ 357us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10446,"prepare.queue_time_us":54193,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":48,"raft.queue_time_us":61046,"raft.run_cpu_time_us":362,"raft.run_wall_time_us":12748,"thread_start_us":354664,"threads_started":2,"wal-append.queue_time_us":471680}
I20260806 03:52:01.681936 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.672353 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.682812 2479 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:01.684271 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:01.684773 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 427549us
I20260806 03:52:01.673767 2478 rpcz_store.cc:276] 0806 03:51:59.895244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:00.840084 (+944840us) service_pool.cc:227] Handling call
0806 03:52:00.840520 (+ 436us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:00.844528 (+ 4008us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:00.844574 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:00.998634 (+154060us) spinlock_profiling.cc:243] Waited 7.18 ms on lock 0x7b2c00000e98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:52:01.028892 (+ 30258us) log.cc:842] Serialized 26 byte log entry
0806 03:52:01.471713 (+442821us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:01.502412 (+ 30699us) log.cc:842] Serialized 14 byte log entry
0806 03:52:01.503404 (+ 992us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:01.503486 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:01.561341 (+ 57855us) spinlock_profiling.cc:243] Waited 3.74 ms on lock 0x7b180000a2a0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:01.672647 (+111306us) raft_consensus.cc:1671] finished
0806 03:52:01.672700 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:01.673045 (+ 345us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4733,"prepare.queue_time_us":31841,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":2582,"raft.queue_time_us":2102,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":16386,"spinlock_wait_cycles":25121408,"thread_start_us":306510,"threads_started":1,"wal-append.queue_time_us":456122}
W20260806 03:52:01.687690 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 168495us
I20260806 03:52:01.690573 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:01.707471 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 177858us
I20260806 03:52:01.708133 2476 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:01.713716 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:01.714231 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:01.717648 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:01.719043 2478 raft_consensus.cc:1261] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:52:01.719297 3584 raft_consensus.cc:2379] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:01.721661 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:01.669878 2332 leader_election.cc:334] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.668s (SENT)
W20260806 03:52:01.724861 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 55081.5us
W20260806 03:52:01.725800 2332 leader_election.cc:334] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.944s (SENT)
W20260806 03:52:01.727880 2332 leader_election.cc:334] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 3.120s (SENT)
I20260806 03:52:01.729310 2476 raft_consensus.cc:3032] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:01.730837 2476 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:01.730362 2332 leader_election.cc:334] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 3.144s (SENT)
I20260806 03:52:01.733210 3594 raft_consensus.cc:1226] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:52:01.734901 5437 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:52:01.735280 5474 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.832s user 0.004s sys 0.007s
I20260806 03:52:01.735561 5437 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:52:01.735939 5474 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.833s user 0.004s sys 0.007s
I20260806 03:52:01.736265 5437 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:01.756340 5607 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.004s sys 0.003s
W20260806 03:52:01.757094 5607 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.746s user 0.005s sys 0.003s
I20260806 03:52:01.758531 5689 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:01.760710 2332 leader_election.cc:334] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 3.451s (SENT)
I20260806 03:52:01.768270 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:01.772589 5624 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:01.773175 5624 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:01.773521 5624 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:01.776430 5625 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:01.777312 5625 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:52:01.776768 2476 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:01.778012 5625 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:01.779349 2476 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:01.787791 5725 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:01.798300 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:01.840378 2377 raft_consensus.cc:1261] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:01.847926 2369 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
W20260806 03:52:01.849844 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 215919us
W20260806 03:52:01.865046 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 104436us
W20260806 03:52:01.863386 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 192424us
W20260806 03:52:01.865520 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 163743us
I20260806 03:52:01.872102 5714 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:01.873112 5714 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.877948 5698 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:01.878414 5725 consensus_queue.cc:1046] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:52:01.878786 5698 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.880043 5607 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:01.879216 2331 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:01.881788 5723 raft_consensus.cc:3032] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:01.884415 5723 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:01.897727 5723 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:01.899159 2534 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:01.885650 5663 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.690s user 0.011s sys 0.007s
I20260806 03:52:01.900823 5663 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.902595 2483 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:01.903776 2534 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:01.904945 5510 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:01.905807 5663 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:01.900116 5430 raft_consensus.cc:3032] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:52:01.906574 5032 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:01.906796 5430 raft_consensus.cc:729] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:01.910892 5430 consensus_queue.cc:260] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.913697 5430 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
W20260806 03:52:01.914629 5706 log.cc:924] Time spent T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 1.012s user 0.004s sys 0.000s
I20260806 03:52:01.883905 5698 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:01.882827 2377 raft_consensus.cc:1261] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260806 03:52:01.926184 5119 consensus_meta.cc:287] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.753s user 0.015s sys 0.006s
I20260806 03:52:01.926868 5119 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:01.929811 5119 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:01.931736 5746 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.369s user 0.001s sys 0.002s
I20260806 03:52:01.932152 2534 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:01.933063 2370 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "32d01287577744949bfebb63c59f3db6"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:52:01.937474 2370 raft_consensus.cc:595] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:01.938246 2370 raft_consensus.cc:603] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:01.939749 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:01.941152 5236 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:52:01.944499 5236 raft_consensus.cc:686] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:01.945916 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 197) took 1044 ms. Trace:
I20260806 03:52:01.946008 5236 consensus_queue.cc:237] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:01.948904 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 62738.4us
I20260806 03:52:01.950078 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:01.952931 2370 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:01.954025 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:01.955431 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:01.956636 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:01.899791 5723 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:01.946491 2482 rpcz_store.cc:276] 0806 03:52:00.901347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:01.070420 (+169073us) service_pool.cc:227] Handling call
0806 03:52:01.160569 (+ 90149us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:01.160836 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:01.200336 (+ 39500us) log.cc:842] Serialized 14 byte log entry
0806 03:52:01.944887 (+744551us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b18003bb8a0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d7c75482c 00007f0d79b7f9dc 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9a155 00007f0d7cabd3d0 00007f0d7cacf576 00007f0d7cae00d7
0806 03:52:01.945149 (+ 262us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:01.945252 (+ 103us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:01.945356 (+ 104us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:01.945494 (+ 138us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:01.945707 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15757,"prepare.queue_time_us":36941,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":80679,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"spinlock_wait_cycles":3372416,"thread_start_us":298460,"threads_started":2,"wal-append.queue_time_us":311155}
I20260806 03:52:02.029661 2370 raft_consensus.cc:2396] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 03:52:02.033123 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.035138 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.036753 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.037926 2368 raft_consensus.cc:2379] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:02.039553 2368 raft_consensus.cc:1226] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:02.051141 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:02.052867 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 120967us
W20260806 03:52:02.060469 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 73073.4us
I20260806 03:52:02.090013 5688 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:02.090912 5533 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:52:02.095435 5581 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.286s user 0.019s sys 0.001s
I20260806 03:52:02.098793 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1819) took 2208 ms. Trace:
I20260806 03:52:02.100807 5755 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:02.105445 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.099334 5437 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:02.117654 5575 consensus_meta.cc:287] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.532s user 0.015s sys 0.000s
I20260806 03:52:02.125234 2368 raft_consensus.cc:1226] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:02.126107 3584 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:52:02.129868 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:02.124410 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:02.220965 3587 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:02.115401 5581 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:02.227666 5744 log.cc:924] Time spent T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.665s user 0.001s sys 0.001s
I20260806 03:52:02.230222 5581 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:02.124389 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.231537 2372 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:02.232911 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:02.233496 2332 connection.cc:232] RPC call timeout handler was delayed by -1.3216161020100117s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2524s for a response
I20260806 03:52:02.105418 2484 rpcz_store.cc:276] 0806 03:51:59.880760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:00.696127 (+815367us) service_pool.cc:227] Handling call
0806 03:52:00.696586 (+ 459us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:00.701630 (+ 5044us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:00.701680 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:00.756171 (+ 54491us) log.cc:842] Serialized 26 byte log entry
0806 03:52:00.884000 (+127829us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:00.903703 (+ 19703us) log.cc:842] Serialized 14 byte log entry
0806 03:52:00.917993 (+ 14290us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:00.918076 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:01.542083 (+624007us) spinlock_profiling.cc:243] Waited 9.41 ms on lock 0x7b18003973c0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d764 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:01.913254 (+371171us) spinlock_profiling.cc:243] Waited 81.2 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:02.063792 (+150538us) raft_consensus.cc:1671] finished
0806 03:52:02.063824 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:02.088640 (+ 24816us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15678,"prepare.queue_time_us":27655,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":95803,"raft.run_cpu_time_us":312,"raft.run_wall_time_us":12074,"spinlock_wait_cycles":208395904,"thread_start_us":127307,"threads_started":1,"wal-append.queue_time_us":145480}
I20260806 03:52:02.144928 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.174736 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:02.128325 5624 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:02.205080 5769 log.cc:924] Time spent T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.066s user 0.001s sys 0.004s
I20260806 03:52:02.209096 5437 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:02.212669 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.133598 2482 raft_consensus.cc:3037] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 5
I20260806 03:52:02.131017 2537 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:02.233685 2372 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:02.234992 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:02.207335 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 79489.2us
I20260806 03:52:02.234491 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 195) took 1321 ms. Trace:
I20260806 03:52:02.234627 3586 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
W20260806 03:52:02.249457 2332 leader_election.cc:334] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.592s (SENT)
I20260806 03:52:02.242782 2577 rpcz_store.cc:276] 0806 03:52:00.913090 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:00.915392 (+ 2302us) service_pool.cc:227] Handling call
0806 03:52:02.233979 (+1318587us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b540004b354. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:02.234250 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1073885568}
I20260806 03:52:02.252727 2332 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:02.254073 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.255846 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.257810 2370 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:02.260152 5624 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:02.264102 2376 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:02.267485 5575 consensus_queue.cc:1046] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:52:02.269591 5575 raft_consensus.cc:3032] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:02.270030 5575 raft_consensus.cc:729] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:02.270738 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:02.271682 2481 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:02.271132 5575 consensus_queue.cc:260] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:02.272367 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:02.273499 2369 raft_consensus.cc:2396] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 7 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:02.292446 2373 raft_consensus.cc:1261] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:02.293025 2479 raft_consensus.cc:1261] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:52:02.293435 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1856) took 1205 ms. Trace:
I20260806 03:52:02.294289 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.295015 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1858) took 1058 ms. Trace:
I20260806 03:52:02.293915 2373 rpcz_store.cc:276] 0806 03:52:01.088551 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:01.169183 (+ 80632us) service_pool.cc:227] Handling call
0806 03:52:02.291248 (+1122065us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:52:02.291589 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:02.293091 (+ 1502us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:02.293333 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":28967808}
I20260806 03:52:02.286085 5624 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:52:02.317857 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:02.321348 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:02.295367 2379 rpcz_store.cc:276] 0806 03:52:01.236663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:01.236857 (+ 194us) service_pool.cc:227] Handling call
0806 03:52:01.279419 (+ 42562us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:01.656833 (+377414us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:01.855167 (+198334us) log.cc:842] Serialized 14 byte log entry
0806 03:52:02.050936 (+195769us) log.cc:842] Serialized 14 byte log entry
0806 03:52:02.051632 (+ 696us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:02.051692 (+ 60us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:02.051759 (+ 67us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:02.294604 (+242845us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b5400059bd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:02.294650 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:02.294795 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":253057,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":131,"raft.queue_time_us":2557,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"spinlock_wait_cycles":866177792,"thread_start_us":370349,"threads_started":2,"wal-append.queue_time_us":379483}
I20260806 03:52:02.325255 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:02.342126 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 214688us
I20260806 03:52:02.350778 5540 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:02.335043 5032 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.023s
I20260806 03:52:02.356058 5624 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:02.356909 5032 raft_consensus.cc:3032] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:02.357518 5032 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:02.357577 5624 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:02.359001 5032 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:02.338447 5575 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:02.370504 2537 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:02.397394 5510 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:52:02.385890 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.401743 2368 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 6: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:02.402333 5639 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:02.403993 3586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 366) took 1433 ms. Trace:
I20260806 03:52:02.404505 3586 rpcz_store.cc:276] 0806 03:52:00.970739 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:00.992800 (+ 22061us) service_pool.cc:227] Handling call
0806 03:52:02.403579 (+1410779us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540070d0d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:02.403869 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2339336064}
I20260806 03:52:02.406500 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 199) took 1501 ms. Trace:
I20260806 03:52:02.406931 2483 rpcz_store.cc:276] 0806 03:52:00.905422 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:01.325462 (+420040us) service_pool.cc:227] Handling call
0806 03:52:01.371127 (+ 45665us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:01.903150 (+532023us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:02.406192 (+503042us) spinlock_profiling.cc:243] Waited 531 ms on lock 0x7b5400350cd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:02.406414 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1221075328}
W20260806 03:52:02.432283 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 185529us
I20260806 03:52:02.378877 5758 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:02.436995 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 68486.4us
I20260806 03:52:02.390265 5624 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:02.411222 5032 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:02.444392 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54212 (request call id 449) took 1270 ms. Trace:
I20260806 03:52:02.450024 2478 rpcz_store.cc:276] 0806 03:52:01.168953 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:01.720339 (+551386us) service_pool.cc:227] Handling call
0806 03:52:02.395371 (+675032us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b54000af0d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4aaf0 00007f0d7cacdf2a 00007f0d81a0ac6e 00007f0d7b101807 00007f0d7b101792 00007f0d7b1016c1 00007f0d7b10163c 00007f0d7b1008f2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:02.438805 (+ 43434us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":810550144}
I20260806 03:52:02.457795 3591 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:02.460072 2534 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:02.463097 2483 raft_consensus.cc:1226] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:52:02.439287 5639 consensus_queue.cc:1035] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20260806 03:52:02.475313 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:02.486552 2537 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:02.499065 2376 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:02.501798 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:02.503134 3584 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260806 03:52:02.507844 2332 leader_election.cc:334] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.973s (SENT)
I20260806 03:52:02.515448 2332 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:02.521559 2372 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:02.522364 2379 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:02.523313 5723 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:02.524010 5723 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:02.527757 3586 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:52:02.522392 5702 consensus_queue.cc:1035] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260806 03:52:02.530752 3591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 378) took 1452 ms. Trace:
I20260806 03:52:02.531287 3591 rpcz_store.cc:276] 0806 03:52:01.078540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:01.078752 (+ 212us) service_pool.cc:227] Handling call
0806 03:52:01.923674 (+844922us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b5800000f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:52:02.412209 (+488535us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:02.458535 (+ 46326us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:02.530451 (+ 71916us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b540070d0d0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:02.530629 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1122639872}
I20260806 03:52:02.533607 5435 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:02.505746 5510 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:52:02.513684 5714 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:02.518556 5780 log.cc:924] Time spent T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260806 03:52:02.542320 5682 consensus_queue.cc:1046] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260806 03:52:02.544318 5717 raft_consensus.cc:3032] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:02.544740 5717 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:02.545780 5717 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:02.547329 5717 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:02.468314 5758 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:02.556454 2482 raft_consensus.cc:2454] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:02.561026 2481 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:02.562037 5723 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:02.565012 2236 catalog_manager.cc:5495] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 4, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 4 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
W20260806 03:52:02.580711 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 64934.6us
W20260806 03:52:02.580001 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 60667.7us
W20260806 03:52:02.582511 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 111148us
I20260806 03:52:02.576623 5625 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:02.593247 2534 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:02.594611 5756 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:02.595069 5756 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:02.595472 5756 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:02.595762 5624 raft_consensus.cc:1050] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
W20260806 03:52:02.596463 5624 raft_consensus.cc:1054] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Unable to promote non-voter cf6042e85bef47c4aac9d064d91501d1: Illegal state: Leader has not yet committed an operation in its own term
I20260806 03:52:02.591264 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:02.603304 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 105697us
I20260806 03:52:02.578749 5510 raft_consensus.cc:686] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:02.585407 5437 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260806 03:52:02.605976 5607 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260806 03:52:02.624243 5510 consensus_queue.cc:237] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:02.628896 5755 consensus_queue.cc:1035] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:02.617151 5435 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:02.617928 2236 catalog_manager.cc:5495] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:02.630296 5435 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:02.631220 2236 catalog_manager.cc:5495] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 3 to 5, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 5 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:02.605693 5624 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260806 03:52:02.641271 5435 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:02.663403 5770 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:02.663740 5625 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:02.666553 3547 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:02.674103 5625 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:02.676637 5625 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:02.676857 5758 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:02.664219 5770 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:02.685823 5770 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:02.678928 2236 catalog_manager.cc:5495] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 4, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 4 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:52:02.694175 2368 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
W20260806 03:52:02.695979 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 109906us
I20260806 03:52:02.697820 2483 raft_consensus.cc:1261] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 03:52:02.655767 5533 consensus_queue.cc:1046] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260806 03:52:02.697291 2236 catalog_manager.cc:5495] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:02.719472 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:02.718466 2236 catalog_manager.cc:5495] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 4, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 4 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:02.712563 5474 raft_consensus.cc:2781] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:02.701334 5689 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.031s
I20260806 03:52:02.764307 5587 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:02.765086 5587 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:02.776275 5685 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:02.770583 5758 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:02.777158 2236 catalog_manager.cc:5495] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 4, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 4 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:02.770291 5474 raft_consensus.cc:686] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:02.781508 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:02.770674 5689 consensus_queue.cc:1225] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer cf6042e85bef47c4aac9d064d91501d1 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260806 03:52:02.805038 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:02.809726 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.814153 2236 catalog_manager.cc:5495] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:02.821050 2376 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:02.821110 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:02.827688 5723 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:02.842065 3587 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.620s user 0.015s sys 0.002s
I20260806 03:52:02.841921 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:02.842484 3587 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:02.843735 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:02.847306 2374 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: [3.5-5.6]
I20260806 03:52:02.847586 5801 raft_consensus.cc:3032] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:02.848146 5801 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:02.849400 5790 log.cc:924] Time spent T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 03:52:02.814056 5430 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:02.850634 5430 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:02.852517 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:02.853152 5587 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:02.853221 2579 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:02.850648 5684 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:02.854480 5684 raft_consensus.cc:686] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:02.854408 5474 consensus_queue.cc:237] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:02.855768 5684 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:02.849184 5801 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:02.851667 2429 leader_election.cc:334] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.927s (SENT)
I20260806 03:52:02.864830 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.867771 2236 catalog_manager.cc:5495] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 4, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 4 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:02.853071 5430 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:02.875450 2236 catalog_manager.cc:5495] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:02.892237 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:02.900697 2480 raft_consensus.cc:1261] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:52:02.814988 5685 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:02.909227 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:02.917208 2579 raft_consensus.cc:2396] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 03:52:02.827823 5758 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:02.925379 2236 catalog_manager.cc:5495] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 6, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 6 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
W20260806 03:52:02.925769 2429 leader_election.cc:334] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.944s (SENT)
I20260806 03:52:02.922331 5801 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:02.927387 2236 catalog_manager.cc:5495] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 5, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 5 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
W20260806 03:52:02.935673 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7165493888314813s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1524s for a response
W20260806 03:52:02.936242 2436 connection.cc:232] RPC call timeout handler was delayed by -2.6904493886977434s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1553s for a response
W20260806 03:52:02.936587 2436 connection.cc:232] RPC call timeout handler was delayed by -2.6652493888977915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15810000000000002s for a response
W20260806 03:52:02.936870 2436 connection.cc:232] RPC call timeout handler was delayed by -2.6364493886940181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1613s for a response
W20260806 03:52:02.937126 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5860493888612837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16690000000000002s for a response
W20260806 03:52:02.937424 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5563493887893856s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17020000000000002s for a response
W20260806 03:52:02.937692 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5212493888102472s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17410000000000003s for a response
W20260806 03:52:02.937978 2436 connection.cc:232] RPC call timeout handler was delayed by -2.4699493888765574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17980000000000002s for a response
W20260806 03:52:02.938256 2436 connection.cc:232] RPC call timeout handler was delayed by -2.4654493888374418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18030000000000002s for a response
W20260806 03:52:02.938537 2436 connection.cc:232] RPC call timeout handler was delayed by -2.4654493888374418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18030000000000002s for a response
W20260806 03:52:02.938874 2436 connection.cc:232] RPC call timeout handler was delayed by -2.4096493888646364s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1865s for a response
W20260806 03:52:02.939180 2436 connection.cc:232] RPC call timeout handler was delayed by -2.4096493888646364s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1865s for a response
W20260806 03:52:02.939497 2436 connection.cc:232] RPC call timeout handler was delayed by -2.3718493888154626s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1907s for a response
W20260806 03:52:02.939765 2436 connection.cc:232] RPC call timeout handler was delayed by -2.3691493887454271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
I20260806 03:52:02.935521 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:02.940855 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:02.922768 5430 consensus_queue.cc:1035] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20260806 03:52:02.917613 5723 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:52:02.945798 2436 connection.cc:232] RPC call timeout handler was delayed by -2.3691493887454271s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.191s for a response
I20260806 03:52:02.945132 2370 raft_consensus.cc:1261] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:52:02.933218 2236 catalog_manager.cc:5495] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
W20260806 03:52:02.951858 2436 connection.cc:232] RPC call timeout handler was delayed by -2.3646493887063116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1915s for a response
W20260806 03:52:02.952471 2436 connection.cc:232] RPC call timeout handler was delayed by -2.353849388891831s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
I20260806 03:52:02.923323 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:02.953403 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:02.954821 5800 log.cc:924] Time spent T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.096s user 0.001s sys 0.000s
I20260806 03:52:02.956197 5794 raft_consensus.cc:1050] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:52:02.956568 3590 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:02.956605 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:02.969848 2436 connection.cc:232] RPC call timeout handler was delayed by -2.353849388891831s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1927s for a response
W20260806 03:52:03.013630 2436 connection.cc:232] RPC call timeout handler was delayed by -2.3385493888054043s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19440000000000002s for a response
W20260806 03:52:02.952224 5575 consensus_meta.cc:287] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.577s user 0.015s sys 0.000s
I20260806 03:52:02.982208 2236 catalog_manager.cc:5495] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 2, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 2 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
W20260806 03:52:03.014082 2436 connection.cc:232] RPC call timeout handler was delayed by -2.3304493888281286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19530000000000003s for a response
W20260806 03:52:03.015318 2436 connection.cc:232] RPC call timeout handler was delayed by -2.2692493887152523s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2021s for a response
W20260806 03:52:03.015715 2436 connection.cc:232] RPC call timeout handler was delayed by -2.2602493888698518s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20310000000000003s for a response
W20260806 03:52:03.016043 2436 connection.cc:232] RPC call timeout handler was delayed by -2.2467493887525052s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2046s for a response
W20260806 03:52:03.016196 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 66505.7us
W20260806 03:52:03.016321 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1954493888188154s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21030000000000004s for a response
W20260806 03:52:03.018467 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1846493887715042s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21150000000000002s for a response
W20260806 03:52:03.018847 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1576493887696415s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21450000000000002s for a response
W20260806 03:52:03.019196 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1558493888005614s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2147s for a response
W20260806 03:52:03.019523 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1513493887614459s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
I20260806 03:52:02.943568 5756 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:03.019840 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1243493887595832s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2182s for a response
W20260806 03:52:03.022217 2436 connection.cc:232] RPC call timeout handler was delayed by -2.1045493888668716s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22040000000000004s for a response
W20260806 03:52:03.022521 2436 connection.cc:232] RPC call timeout handler was delayed by -2.0811493888031691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
I20260806 03:52:02.980942 5723 raft_consensus.cc:686] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
W20260806 03:52:03.022773 2436 connection.cc:232] RPC call timeout handler was delayed by -2.0694493888877332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
W20260806 03:52:03.025245 5793 log.cc:924] Time spent T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.100s user 0.001s sys 0.001s
W20260806 03:52:03.012816 5794 raft_consensus.cc:1054] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Unable to promote non-voter cf6042e85bef47c4aac9d064d91501d1: Illegal state: Leader has not yet committed an operation in its own term
I20260806 03:52:03.037345 2579 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:03.036036 2436 connection.cc:232] RPC call timeout handler was delayed by -2.0694493888877332s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2243s for a response
W20260806 03:52:03.038164 2436 connection.cc:232] RPC call timeout handler was delayed by -1.6653493887279183s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26920000000000005s for a response
W20260806 03:52:03.038560 2436 connection.cc:232] RPC call timeout handler was delayed by -1.551049388712272s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2819s for a response
W20260806 03:52:03.044482 5795 log.cc:924] Time spent T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.112s user 0.002s sys 0.000s
I20260806 03:52:03.045291 5758 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:03.047407 5756 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:03.052525 5685 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:03.038897 2436 connection.cc:232] RPC call timeout handler was delayed by -1.2927493888419122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3106s for a response
W20260806 03:52:03.055668 2436 connection.cc:232] RPC call timeout handler was delayed by -1.1955493888817728s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3214s for a response
W20260806 03:52:03.056160 2436 connection.cc:232] RPC call timeout handler was delayed by -1.0650493886787444s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33590000000000003s for a response
W20260806 03:52:03.056582 2436 connection.cc:232] RPC call timeout handler was delayed by -1.0623493888415396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33620000000000005s for a response
I20260806 03:52:03.022778 2236 catalog_manager.cc:5495] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260806 03:52:03.080295 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 63303.4us
W20260806 03:52:03.085757 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 99195.7us
I20260806 03:52:03.096977 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:03.100775 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:03.097913 2372 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:03.091871 2236 catalog_manager.cc:5495] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 3 to 6. New cstate: current_term: 6 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
W20260806 03:52:03.121739 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 52384.5us
I20260806 03:52:03.124802 5758 consensus_queue.cc:1046] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:03.125979 5758 consensus_queue.cc:1225] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer dff94100ffc5485f92cbe6a85addf0b2 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260806 03:52:03.128373 5430 raft_consensus.cc:3032] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:03.128775 5430 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:03.129865 5430 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:03.131745 5430 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:03.089079 3586 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:03.123633 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:03.143229 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 43795.5us
I20260806 03:52:03.145282 5829 consensus_queue.cc:1035] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:52:03.149902 5689 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260806 03:52:03.141525 5816 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20260806 03:52:03.144420 2236 catalog_manager.cc:5495] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 4, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 4 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
W20260806 03:52:03.181770 2429 leader_election.cc:334] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.115s (SENT)
W20260806 03:52:03.172721 5717 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.625s user 0.010s sys 0.011s
I20260806 03:52:03.184054 2236 catalog_manager.cc:5495] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 4, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 4 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260806 03:52:03.193665 3591 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:03.195168 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:03.195917 2584 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:03.184391 5723 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:03.199139 2436 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:03.183100 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:03.206986 2429 leader_election.cc:334] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.145s (SENT)
I20260806 03:52:03.209731 2583 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:03.212899 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 241065us
W20260806 03:52:03.212903 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 64106.6us
I20260806 03:52:03.228530 2331 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:03.199146 2236 catalog_manager.cc:5495] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 3, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 3 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:03.206552 5639 raft_consensus.cc:3032] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:03.243484 2236 catalog_manager.cc:5495] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 2 to 5, leader changed from cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 5 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260806 03:52:03.259205 2429 leader_election.cc:334] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.152s (SENT)
I20260806 03:52:03.281867 2578 raft_consensus.cc:1261] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:52:03.254238 2236 catalog_manager.cc:5495] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:03.214623 5822 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:03.254093 5639 raft_consensus.cc:729] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:03.240062 5794 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:52:03.332250 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 179814us
I20260806 03:52:03.344524 2584 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:03.382922 5639 consensus_queue.cc:260] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:03.384706 5639 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:03.385707 2534 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:03.376492 5794 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:03.390352 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:03.390869 5794 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:03.375313 2579 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:03.344677 5390 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:03.396497 5390 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:03.402509 5830 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:03.403072 5830 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:03.403462 5830 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:03.359558 5815 log.cc:924] Time spent T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.334s user 0.000s sys 0.002s
I20260806 03:52:03.407935 5390 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:03.412187 5824 log.cc:924] Time spent T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.202s user 0.001s sys 0.000s
I20260806 03:52:03.412204 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:03.413166 2375 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 1.
W20260806 03:52:03.360283 5812 log.cc:924] Time spent T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20260806 03:52:03.392669 2538 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:03.416359 2375 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 03:52:03.416482 5540 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:03.421020 5820 log.cc:924] Time spent T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20260806 03:52:03.421481 5435 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:03.424289 5390 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:52:03.361260 5825 log.cc:924] Time spent T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20260806 03:52:03.432602 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 261222us
I20260806 03:52:03.433941 2332 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:03.435096 5760 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:52:03.437705 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:03.438710 2574 raft_consensus.cc:2379] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:03.440631 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:03.442967 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:03.445504 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.446271 2578 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:03.447283 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.452070 2578 raft_consensus.cc:2379] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:03.439752 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.455147 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:03.459995 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 305547us
I20260806 03:52:03.457623 2583 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:03.462512 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.467990 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:03.470304 2481 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:03.470561 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:03.473048 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:03.474622 2583 raft_consensus.cc:2396] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:52:03.501202 2476 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
I20260806 03:52:03.510012 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.511515 2578 raft_consensus.cc:2396] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260806 03:52:03.520208 3547 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:03.546535 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.547804 2574 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 2.
W20260806 03:52:03.548990 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 115307us
W20260806 03:52:03.554781 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 71633.7us
W20260806 03:52:03.593736 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 83440.9us
I20260806 03:52:03.569756 5717 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:52:03.570376 5822 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:03.613226 2581 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:03.614749 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.601415 5717 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:03.668967 5430 consensus_meta.cc:287] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.537s user 0.011s sys 0.006s
I20260806 03:52:03.648928 5717 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:03.617486 5822 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:03.877961 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:03.880393 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 275875us
I20260806 03:52:03.881516 2436 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:03.882609 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 362495us
I20260806 03:52:03.877776 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.887387 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.888243 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.888799 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:03.890043 2332 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:03.900988 5607 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:52:03.929838 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 336871us
I20260806 03:52:03.904373 5829 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:03.926194 5830 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.522s user 0.008s sys 0.008s
I20260806 03:52:03.904605 5760 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:03.937263 2331 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:03.974877 2480 raft_consensus.cc:1261] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:52:03.933310 5607 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:03.996619 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 68745us
W20260806 03:52:03.999857 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 66680.8us
W20260806 03:52:04.002887 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 72150.7us
W20260806 03:52:04.057672 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 90787us
W20260806 03:52:04.066937 2429 leader_election.cc:334] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.629s (SENT)
I20260806 03:52:04.007493 5607 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:04.000558 5853 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:03.958042 5830 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:04.128031 2332 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:03.998907 5829 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:04.147253 2579 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:04.116631 5853 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:52:04.130555 5858 log.cc:924] Time spent T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
W20260806 03:52:04.177738 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 147267us
I20260806 03:52:04.145573 2538 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:04.167598 5853 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:04.247493 5849 consensus_queue.cc:1035] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.163s
I20260806 03:52:04.191942 5822 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:04.309327 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 73407.2us
W20260806 03:52:04.307766 5832 log.cc:924] Time spent T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.171s user 0.001s sys 0.000s
I20260806 03:52:04.339110 2481 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260806 03:52:04.340052 2429 leader_election.cc:334] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.809s (SENT)
W20260806 03:52:04.333922 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 188471us
I20260806 03:52:04.302521 5723 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:03.995718 5760 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:04.361202 2374 raft_consensus.cc:1261] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:52:04.333906 5822 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:04.412299 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.021s sys 0.063s
W20260806 03:52:04.418953 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 71828.7us
W20260806 03:52:04.433565 2577 consensus_meta.cc:287] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.760s user 0.013s sys 0.000s
I20260806 03:52:04.441819 2332 leader_election.cc:398] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5.
W20260806 03:52:04.428503 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.523s user 0.021s sys 0.063s
I20260806 03:52:04.443008 2577 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
W20260806 03:52:04.453364 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 110687us
I20260806 03:52:04.456830 2538 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:04.447593 5723 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:04.427904 5900 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:52:04.491242 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 139426us
W20260806 03:52:04.486004 3545 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.646s user 0.009s sys 0.022s
W20260806 03:52:04.501112 3545 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.661s user 0.009s sys 0.023s
W20260806 03:52:04.501538 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 692096us
W20260806 03:52:04.509510 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 628078us
W20260806 03:52:04.480859 5639 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.090s user 0.023s sys 0.001s
W20260806 03:52:04.516594 3545 leader_election.cc:334] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Timed out: RequestConsensusVote RPC to 127.0.183.3:46407 timed out after 2.103s (SENT)
I20260806 03:52:04.533845 2576 raft_consensus.cc:2396] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260806 03:52:04.484364 5430 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:04.542071 3545 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:04.440426 5822 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:04.546299 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:04.551321 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:04.511441 5830 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:04.540408 5846 log.cc:924] Time spent T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.364s user 0.001s sys 0.001s
I20260806 03:52:04.555935 2377 raft_consensus.cc:2396] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:52:04.576289 2436 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:04.569707 2332 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:04.552947 5901 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:04.583227 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1916) took 1098 ms. Trace:
I20260806 03:52:04.583751 2372 rpcz_store.cc:276] 0806 03:52:03.485233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.506354 (+ 21121us) service_pool.cc:227] Handling call
0806 03:52:03.532642 (+ 26288us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:03.730866 (+198224us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:03.730906 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:03.897672 (+166766us) log.cc:842] Serialized 26 byte log entry
0806 03:52:04.002090 (+104418us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:04.002147 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:04.002246 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:04.582957 (+580711us) raft_consensus.cc:1671] finished
0806 03:52:04.582984 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:04.583148 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3124,"raft.queue_time_us":60642,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":19330,"thread_start_us":238163,"threads_started":2,"wal-append.queue_time_us":651197}
I20260806 03:52:04.588964 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2157) took 1157 ms. Trace:
I20260806 03:52:04.589592 2579 rpcz_store.cc:276] 0806 03:52:03.431648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.462159 (+ 30511us) service_pool.cc:227] Handling call
0806 03:52:04.588649 (+1126490us) spinlock_profiling.cc:243] Waited 682 ms on lock 0x7b540024c2d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:04.588868 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1568632192}
W20260806 03:52:04.591312 5867 log.cc:924] Time spent T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.281s user 0.001s sys 0.000s
I20260806 03:52:04.591682 5733 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:04.593029 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 247) took 1133 ms. Trace:
W20260806 03:52:04.593474 5864 log.cc:924] Time spent T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.151s user 0.000s sys 0.001s
W20260806 03:52:04.594821 5839 log.cc:924] Time spent T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.288s user 0.000s sys 0.001s
I20260806 03:52:04.593513 2368 rpcz_store.cc:276] 0806 03:52:03.459566 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.459710 (+ 144us) service_pool.cc:227] Handling call
0806 03:52:03.500528 (+ 40818us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:03.503572 (+ 3044us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:03.503624 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:03.553218 (+ 49594us) log.cc:842] Serialized 52 byte log entry
0806 03:52:03.889618 (+336400us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:03.891168 (+ 1550us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:03.891245 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:04.592681 (+701436us) raft_consensus.cc:1671] finished
0806 03:52:04.592712 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:04.592950 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2404,"raft.queue_time_us":2571,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":13254,"thread_start_us":281505,"threads_started":1,"wal-append.queue_time_us":573775}
W20260806 03:52:04.596012 5838 log.cc:924] Time spent T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.149s user 0.000s sys 0.000s
W20260806 03:52:04.610395 5877 log.cc:924] Time spent T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
W20260806 03:52:04.623880 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 107322us
I20260806 03:52:04.640810 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2156) took 1251 ms. Trace:
I20260806 03:52:04.641342 2575 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260806 03:52:04.641546 2577 rpcz_store.cc:276] 0806 03:52:03.389798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.389971 (+ 173us) service_pool.cc:227] Handling call
0806 03:52:04.640485 (+1250514us) spinlock_profiling.cc:243] Waited 282 ms on lock 0x7b5400188dd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:04.640710 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":647892352}
W20260806 03:52:04.644994 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 78250.2us
I20260806 03:52:04.480564 5723 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:04.558794 2370 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:04.667816 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 150303us
W20260806 03:52:04.676816 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 107231us
W20260806 03:52:04.679893 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 203038us
I20260806 03:52:04.684886 2436 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:04.690366 2332 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:04.668413 5794 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.218s user 0.015s sys 0.002s
I20260806 03:52:04.693178 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 205) took 1248 ms. Trace:
I20260806 03:52:04.578009 5829 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:04.704953 2537 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:04.679406 5900 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:04.700821 2583 rpcz_store.cc:276] 0806 03:52:03.433899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.487070 (+ 53171us) service_pool.cc:227] Handling call
0806 03:52:03.508209 (+ 21139us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:03.570217 (+ 62008us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:52:03.597993 (+ 27776us) log.cc:842] Serialized 14 byte log entry
0806 03:52:04.452033 (+854040us) spinlock_profiling.cc:243] Waited 952 us on lock 0x7b18003e63e0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b7f345 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9ce22 00007f0d7c7465bb 00007f0d7c74c063 00007f0d7c74c57b 00007f0d7cacf88a
0806 03:52:04.567704 (+115671us) log.cc:842] Serialized 14 byte log entry
0806 03:52:04.682195 (+114491us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:04.682263 (+ 68us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:04.682321 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:04.682413 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:04.682566 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":116452,"prepare.queue_time_us":18600,"prepare.run_cpu_time_us":107,"prepare.run_wall_time_us":111,"raft.queue_time_us":186810,"raft.run_cpu_time_us":301,"raft.run_wall_time_us":3717,"spinlock_wait_cycles":2190208,"thread_start_us":92504,"threads_started":1}
I20260806 03:52:04.682507 5822 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:04.679512 5770 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:04.752717 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 62353.7us
W20260806 03:52:04.755151 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 50317.2us
W20260806 03:52:04.756282 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 76848.8us
W20260806 03:52:04.743705 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.004s sys 0.016s
W20260806 03:52:04.756990 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.683s user 0.004s sys 0.016s
W20260806 03:52:04.757326 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 683408us
I20260806 03:52:04.758020 2331 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:04.712757 5879 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:04.764284 2537 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:04.765120 2331 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:04.755251 5760 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:04.767110 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 82343.4us
I20260806 03:52:04.721698 5913 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:04.780905 2331 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:04.710445 5794 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:04.764227 5733 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:04.805361 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 45171.8us
I20260806 03:52:04.771027 5879 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:04.795691 2436 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:04.807212 2537 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:04.759476 5829 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:04.744670 5770 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:04.805405 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.627s user 0.003s sys 0.032s
W20260806 03:52:04.820847 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.661s user 0.004s sys 0.033s
I20260806 03:52:04.764966 5919 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:04.816515 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1882) took 1371 ms. Trace:
I20260806 03:52:04.821736 2332 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:04.818101 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:04.825721 2474 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:04.818980 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:04.792621 5760 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:04.823854 2484 rpcz_store.cc:276] 0806 03:52:03.445554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.445731 (+ 177us) service_pool.cc:227] Handling call
0806 03:52:03.446197 (+ 466us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:03.447145 (+ 948us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:03.447186 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:03.447989 (+ 803us) log.cc:842] Serialized 52 byte log entry
0806 03:52:03.584576 (+136587us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:03.868964 (+284388us) log.cc:842] Serialized 14 byte log entry
0806 03:52:03.955406 (+ 86442us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:03.955526 (+ 120us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:04.586559 (+631033us) spinlock_profiling.cc:243] Waited 5.6 ms on lock 0x7b1800397d80. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:04.810033 (+223474us) raft_consensus.cc:1671] finished
0806 03:52:04.816261 (+ 6228us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:04.816443 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32605,"prepare.queue_time_us":540601,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":585559,"raft.run_cpu_time_us":415,"raft.run_wall_time_us":26795,"spinlock_wait_cycles":12877184,"thread_start_us":455665,"threads_started":3,"wal-append.queue_time_us":667157}
W20260806 03:52:04.834460 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 394647us
I20260806 03:52:04.837793 2477 raft_consensus.cc:2396] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260806 03:52:04.839623 2583 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
W20260806 03:52:04.839810 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 63670.2us
I20260806 03:52:04.816601 5879 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:04.810648 5872 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:04.840101 2583 pending_rounds.cc:76] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 5
I20260806 03:52:04.823138 5919 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:04.808619 5733 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:04.860481 2583 pending_rounds.cc:97] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260806 03:52:04.866791 2583 raft_consensus.cc:2866] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:04.864614 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 202) took 1431 ms. Trace:
I20260806 03:52:04.830042 5032 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:04.867589 2576 rpcz_store.cc:276] 0806 03:52:03.432841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.440196 (+ 7355us) service_pool.cc:227] Handling call
0806 03:52:04.853869 (+1413673us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540040fbd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:04.864504 (+ 10635us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2491061120}
I20260806 03:52:04.838681 5760 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
W20260806 03:52:04.876320 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 84770.6us
W20260806 03:52:04.882839 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 44969.4us
I20260806 03:52:04.884181 3545 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:04.858285 5901 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:52:04.885653 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 54179.1us
I20260806 03:52:04.881493 2331 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:04.851184 5874 raft_consensus.cc:2741] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260806 03:52:04.792069 5913 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:04.892923 5917 consensus_queue.cc:1035] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.125s
I20260806 03:52:04.893890 5733 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:04.895102 5770 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:04.869289 5919 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:04.869566 5926 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:04.898085 2436 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:04.904337 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 44230us
W20260806 03:52:04.893885 5717 consensus_meta.cc:287] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.030s user 0.018s sys 0.000s
I20260806 03:52:04.927937 5760 raft_consensus.cc:686] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:04.929134 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:04.906672 5875 log.cc:924] Time spent T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.211s user 0.000s sys 0.002s
W20260806 03:52:04.931493 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 51993.2us
W20260806 03:52:04.945530 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 124587us
I20260806 03:52:04.954444 5794 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:04.955284 5794 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:04.955797 5794 raft_consensus.cc:686] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:04.950640 5032 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:04.958935 5032 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:04.921327 5926 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:04.960639 5926 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:04.961349 5032 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:04.961473 5848 log.cc:924] Time spent T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.783s user 0.002s sys 0.002s
I20260806 03:52:04.961018 5760 consensus_queue.cc:237] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:04.962055 5931 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:04.930204 5723 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:04.898639 5851 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:04.964943 5851 raft_consensus.cc:686] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:04.951855 5717 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:04.966038 5851 consensus_queue.cc:237] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:04.904844 5901 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:52:04.969015 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 71011.5us
I20260806 03:52:04.969965 2436 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:04.971381 2436 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:04.973011 5912 consensus_queue.cc:1035] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
W20260806 03:52:04.960206 5840 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.875s user 0.002s sys 0.000s
I20260806 03:52:04.970109 5901 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:04.956887 5794 consensus_queue.cc:237] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:04.919821 5770 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:04.973847 5755 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:04.981295 5874 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:04.982024 5882 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:04.976423 2332 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:04.986306 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 102265us
I20260806 03:52:05.005564 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:04.985872 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.006357 2577 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:05.006433 2374 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:05.011148 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.014528 5928 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:05.015141 5928 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.017697 5928 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:05.018433 5685 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:05.018847 5897 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.090s
I20260806 03:52:05.019096 5685 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.000422 5770 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:05.022710 2572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 203) took 1589 ms. Trace:
I20260806 03:52:05.022908 5770 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:05.028712 5880 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:05.028402 2572 rpcz_store.cc:276] 0806 03:52:03.433226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.441437 (+ 8211us) service_pool.cc:227] Handling call
0806 03:52:03.441818 (+ 381us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:03.442096 (+ 278us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:03.442120 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:03.442642 (+ 522us) log.cc:842] Serialized 26 byte log entry
0806 03:52:03.585835 (+143193us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:03.585888 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:03.586001 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:05.020575 (+1434574us) raft_consensus.cc:1671] finished
0806 03:52:05.020599 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:05.022505 (+ 1906us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":142764,"threads_started":1,"wal-append.queue_time_us":1439779}
I20260806 03:52:05.031265 5913 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:05.028309 5935 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:05.031961 5119 consensus_queue.cc:1046] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:05.033215 5119 raft_consensus.cc:3032] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:05.033550 5685 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:05.034221 5853 consensus_meta.cc:287] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.720s user 0.009s sys 0.004s
I20260806 03:52:05.034765 5853 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:05.037084 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 65808.2us
I20260806 03:52:05.038453 5926 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:05.041942 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 56839us
I20260806 03:52:04.996169 5717 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:05.048539 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.045508 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:05.049337 2374 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 3
I20260806 03:52:05.043118 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2155) took 1731 ms. Trace:
I20260806 03:52:05.049321 2369 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
W20260806 03:52:05.056622 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 80313.4us
I20260806 03:52:05.052770 2575 rpcz_store.cc:276] 0806 03:52:03.311932 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.368385 (+ 56453us) service_pool.cc:227] Handling call
0806 03:52:04.640715 (+1272330us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:04.641814 (+ 1099us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:05.042830 (+401016us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b5400188dd0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:05.043031 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2787692928}
I20260806 03:52:05.054833 2534 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:05.077787 2332 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:05.033587 5119 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:05.078305 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:05.056695 5723 raft_consensus.cc:2726] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:05.099571 3591 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 03:52:05.100859 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:05.094622 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:05.096554 2331 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:05.097508 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:05.102385 2476 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:05.097993 5119 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.099184 5894 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:05.099469 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:05.102567 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.089875 3542 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:05.077406 5698 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:05.111398 3590 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:05.134454 2436 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:05.121300 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 57696.9us
I20260806 03:52:05.124540 2370 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:05.176429 3594 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260806 03:52:05.176738 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 53542.5us
W20260806 03:52:05.177731 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 76420.9us
W20260806 03:52:05.176384 5607 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.057s user 0.016s sys 0.001s
I20260806 03:52:05.182832 5607 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:05.132680 5850 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:05.184423 5850 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:05.184808 5850 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:05.185235 5607 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:05.132730 5916 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:05.110751 5894 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:05.186455 5894 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:05.132385 5936 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.092s
W20260806 03:52:05.188923 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 99205.7us
I20260806 03:52:05.188735 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:05.024641 5755 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:05.188728 2376 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:05.214839 2534 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:05.222905 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:05.223835 2577 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:52:05.228644 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 50258.9us
I20260806 03:52:05.231475 5853 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:05.133925 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:52:05.235662 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 60825.2us
I20260806 03:52:05.134473 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:05.239980 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:05.241111 2581 raft_consensus.cc:2396] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260806 03:52:05.243176 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:05.244092 2584 raft_consensus.cc:2379] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:05.245173 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:05.245904 3592 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:05.225919 5913 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:05.247020 5913 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:05.247406 5913 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:05.220709 5936 consensus_queue.cc:1225] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
W20260806 03:52:05.278749 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 45475.5us
I20260806 03:52:05.269059 2474 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:05.224545 5880 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:05.291818 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:05.308795 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:05.236418 5119 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:05.336534 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:05.352671 2583 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:05.261078 5723 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.100s
W20260806 03:52:05.354622 5952 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.181s user 0.000s sys 0.003s
I20260806 03:52:05.356122 3585 raft_consensus.cc:2379] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:05.176900 5698 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.344900 2537 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:05.335880 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:05.252378 5926 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:05.337267 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:05.358280 2332 connection.cc:232] RPC call timeout handler was delayed by -2.8932138360105455s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1633s for a response
W20260806 03:52:05.370005 2332 connection.cc:232] RPC call timeout handler was delayed by -2.7564138360321522s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
I20260806 03:52:05.280174 5942 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:05.364362 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:05.369460 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 257469us
W20260806 03:52:05.370404 2332 connection.cc:232] RPC call timeout handler was delayed by -2.7015138359274715s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
I20260806 03:52:05.381916 2584 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:05.365875 3545 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:05.374202 2577 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:52:05.385733 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1927) took 1059 ms. Trace:
I20260806 03:52:05.386318 2376 rpcz_store.cc:276] 0806 03:52:04.325922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:04.326941 (+ 1019us) service_pool.cc:227] Handling call
0806 03:52:04.327426 (+ 485us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:05.224814 (+897388us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:05.385399 (+160585us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54001eb4d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:05.385646 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1980336384}
W20260806 03:52:05.392195 2332 connection.cc:232] RPC call timeout handler was delayed by -1.6944138361141086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2965s for a response
W20260806 03:52:05.392619 2332 connection.cc:232] RPC call timeout handler was delayed by -1.6674138361122459s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29950000000000004s for a response
W20260806 03:52:05.392939 2332 connection.cc:232] RPC call timeout handler was delayed by -1.6482255579903722s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1502s for a response
I20260806 03:52:05.393128 3591 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:05.393293 2332 connection.cc:232] RPC call timeout handler was delayed by -1.4223255580291152s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1753s for a response
W20260806 03:52:05.393822 2332 connection.cc:232] RPC call timeout handler was delayed by -1.273825558135286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
W20260806 03:52:05.394193 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0533255580812693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2163s for a response
W20260806 03:52:05.394572 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0182255581021309s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20260806 03:52:05.400327 5888 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election won for term 7
W20260806 03:52:05.401402 5951 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20260806 03:52:05.401814 5685 consensus_queue.cc:1046] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:05.399072 2476 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:52:05.412735 2374 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:52:05.414206 2583 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
W20260806 03:52:05.421905 2332 leader_election.cc:334] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.502s (SENT)
I20260806 03:52:05.376924 5937 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:05.433202 2331 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:05.435173 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.437626 2332 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:05.403105 5685 raft_consensus.cc:3032] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:05.243211 5933 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.182s
I20260806 03:52:05.450346 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.447135 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.455125 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.456220 2436 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:05.458500 2375 raft_consensus.cc:1226] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:05.403119 5916 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:05.464628 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:05.466754 2584 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:05.473398 2372 raft_consensus.cc:2354] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:05.424855 5888 raft_consensus.cc:686] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:05.478391 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 109179us
I20260806 03:52:05.436303 5937 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:05.479060 2369 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:52:05.488176 2369 raft_consensus.cc:595] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:05.488590 2369 raft_consensus.cc:603] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:05.459748 5760 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:05.458114 5682 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:05.452919 5685 raft_consensus.cc:729] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:05.489897 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 56837.5us
W20260806 03:52:05.494434 5947 log.cc:924] Time spent T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
W20260806 03:52:05.494594 5879 consensus_meta.cc:287] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.625s user 0.009s sys 0.004s
W20260806 03:52:05.496594 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 130796us
I20260806 03:52:05.461302 5430 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:05.384284 5880 consensus_queue.cc:1046] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.095s
I20260806 03:52:05.391389 5942 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.499009 2584 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 03:52:05.581905 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 160059us
I20260806 03:52:05.567508 2331 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:05.590114 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 200933us
W20260806 03:52:05.682435 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 115039us
I20260806 03:52:05.678690 3545 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:05.685745 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1883) took 2226 ms. Trace:
W20260806 03:52:05.494778 5950 log.cc:924] Time spent T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
I20260806 03:52:05.484575 5907 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:05.491355 5937 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:05.609504 5882 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:05.705803 5894 consensus_meta.cc:287] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.519s user 0.011s sys 0.000s
I20260806 03:52:05.704063 5964 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:05.681808 5879 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.585428 5760 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:05.711820 3545 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:05.712430 2332 leader_election.cc:334] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.633s (SENT)
I20260806 03:52:05.766063 2332 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:05.474764 5916 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:05.681507 5770 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.653s user 0.013s sys 0.000s
I20260806 03:52:05.717613 5685 consensus_queue.cc:260] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:05.703437 5850 consensus_meta.cc:287] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.518s user 0.009s sys 0.002s
I20260806 03:52:05.675931 5682 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:05.689764 5755 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:05.689620 2483 rpcz_store.cc:276] 0806 03:52:03.452858 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.453058 (+ 200us) service_pool.cc:227] Handling call
0806 03:52:03.458819 (+ 5761us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:03.459090 (+ 271us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:03.580314 (+121224us) log.cc:842] Serialized 14 byte log entry
0806 03:52:05.678577 (+2098263us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b1800397fc0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d7c75482c 00007f0d79b7f9dc 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9a155 00007f0d7cabd3d0 00007f0d7cacf576 00007f0d7cae00d7
0806 03:52:05.678702 (+ 125us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:05.678753 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:05.678809 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:05.678895 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:05.679058 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6249,"prepare.queue_time_us":513502,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"raft.queue_time_us":591392,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":28996,"spinlock_wait_cycles":40651520,"thread_start_us":582211,"threads_started":3,"wal-append.queue_time_us":760865,"wal-append.run_cpu_time_us":2389,"wal-append.run_wall_time_us":1265907}
I20260806 03:52:05.780114 5940 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:05.687039 5430 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.782222 5942 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:05.783339 5430 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:05.783660 5954 log.cc:924] Time spent T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.104s user 0.000s sys 0.000s
I20260806 03:52:05.783922 5965 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:05.784554 5965 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.780831 5940 raft_consensus.cc:686] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:05.786969 5965 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:05.786459 5940 consensus_queue.cc:237] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:05.764724 5964 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:05.767648 5947 log.cc:924] Time spent T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260806 03:52:05.755185 5607 consensus_queue.cc:1035] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.046s
W20260806 03:52:05.806629 5953 log.cc:924] Time spent T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.408s user 0.001s sys 0.000s
I20260806 03:52:05.807185 5966 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:05.807813 5966 raft_consensus.cc:686] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:05.808991 5966 consensus_queue.cc:237] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:05.712431 5888 consensus_queue.cc:237] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:05.792096 5850 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:05.790684 5698 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:05.813002 5850 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:05.813853 5698 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:52:05.814381 5698 raft_consensus.cc:686] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:05.761051 5894 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.774258 5959 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:52:05.818087 5894 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:05.760514 5882 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.820683 2572 raft_consensus.cc:2354] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
W20260806 03:52:05.822359 5956 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20260806 03:52:05.822486 5879 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:05.822929 5882 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:05.775410 5770 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:05.825050 5770 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:05.829010 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 129769us
W20260806 03:52:05.846108 2332 leader_election.cc:334] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.753s (SENT)
I20260806 03:52:05.846655 2332 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:05.840811 5919 consensus_meta.cc:287] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.903s user 0.013s sys 0.000s
I20260806 03:52:05.844889 3591 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
W20260806 03:52:05.842264 5913 consensus_meta.cc:287] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.593s user 0.010s sys 0.003s
I20260806 03:52:05.799815 5682 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.772862 5760 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.815392 5698 consensus_queue.cc:237] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.862672 5770 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:52:05.890179 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48214.1us
I20260806 03:52:05.863435 5913 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.897284 5913 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:05.898355 3586 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:05.855074 5919 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:05.900982 5919 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:05.903023 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 191313us
I20260806 03:52:05.908622 2436 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:05.888644 5958 log.cc:924] Time spent T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.168s user 0.001s sys 0.000s
I20260806 03:52:05.888756 5932 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:05.924539 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:05.887949 5916 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:05.934707 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:05.876044 5937 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:05.937834 2369 raft_consensus.cc:1226] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:05.972645 5937 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:05.944243 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:05.974476 2483 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:05.945753 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:05.948369 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:05.951764 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 69709.9us
W20260806 03:52:05.955961 5960 log.cc:924] Time spent T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20260806 03:52:05.959434 5682 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:05.961546 5870 log.cc:924] Time spent T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 1.090s user 0.007s sys 0.012s
I20260806 03:52:05.929911 5983 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:52:05.967025 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:05.979151 3592 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:05.969446 5760 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:05.934710 2476 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:05.994609 2436 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:05.987656 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:06.006183 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:06.009953 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:06.010994 2369 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 7
I20260806 03:52:06.023123 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:06.023909 2483 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:52:06.022114 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:06.029662 3547 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:06.017274 5983 raft_consensus.cc:2726] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:52:06.035218 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 207162us
W20260806 03:52:06.037897 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 354389us
I20260806 03:52:06.043072 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:06.043812 2577 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:06.043659 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:06.046633 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 152274us
I20260806 03:52:06.047169 3545 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:06.048550 2436 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:06.030182 5684 raft_consensus.cc:2741] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 03:52:06.091152 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:06.092223 2581 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:06.039108 5972 raft_consensus.cc:2781] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:06.097015 2584 raft_consensus.cc:1261] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260806 03:52:06.035197 5878 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260806 03:52:06.111332 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:52:06.122306 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 87639.8us
I20260806 03:52:06.193197 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:06.110009 5972 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:06.096946 5880 raft_consensus.cc:2741] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
I20260806 03:52:06.095687 5981 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:06.199565 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:06.199982 5981 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:52:06.138315 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48168.4us
I20260806 03:52:06.175838 5945 raft_consensus.cc:1050] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:52:06.176218 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:06.123596 5918 log.cc:924] Time spent T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 1.251s user 0.007s sys 0.007s
I20260806 03:52:06.186168 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:06.207645 2332 leader_election.cc:334] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.202s (SENT)
I20260806 03:52:06.198392 2577 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:06.190656 2369 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:06.124605 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:06.213991 2584 raft_consensus.cc:1261] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:52:06.215261 2574 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:06.218926 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:06.225924 2369 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:06.209084 3592 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:06.231251 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:06.234185 2332 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:06.200559 5972 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:52:06.235639 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 187428us
I20260806 03:52:06.241091 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:06.241746 2372 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:06.200452 5981 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:06.244320 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:06.247952 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:06.249801 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 122123us
I20260806 03:52:06.244755 2534 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:06.263814 2429 leader_election.cc:398] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:06.264240 2429 leader_election.cc:401] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260806 03:52:06.266110 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58533us
I20260806 03:52:06.265362 2436 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:06.267724 2332 leader_election.cc:334] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.501s (SENT)
I20260806 03:52:06.267854 2537 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:06.228592 5755 consensus_queue.cc:1046] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260806 03:52:06.269145 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:06.270933 5980 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:06.277423 5682 consensus_queue.cc:1035] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:06.278406 3542 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:06.279629 5932 consensus_queue.cc:1035] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:52:06.281169 5932 consensus_queue.cc:1225] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 03:52:06.281797 3594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 390) took 1201 ms. Trace:
I20260806 03:52:06.282274 3594 rpcz_store.cc:276] 0806 03:52:05.080395 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:05.093755 (+ 13360us) service_pool.cc:227] Handling call
0806 03:52:05.171651 (+ 77896us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:05.181289 (+ 9638us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:06.281586 (+1100297us) spinlock_profiling.cc:243] Waited 39.6 ms on lock 0x7b54006ff9d0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:06.281729 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":91038080}
I20260806 03:52:06.287276 2537 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:06.253515 5940 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:06.294251 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2160) took 1289 ms. Trace:
I20260806 03:52:06.260509 5937 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:06.304641 2332 leader_election.cc:334] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.995s (SENT)
I20260806 03:52:06.279915 5682 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:06.310312 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:06.309281 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 55466us
I20260806 03:52:06.300061 5983 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:06.294827 2576 rpcz_store.cc:276] 0806 03:52:05.004827 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:05.011106 (+ 6279us) service_pool.cc:227] Handling call
0806 03:52:05.023672 (+ 12566us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:05.048735 (+ 25063us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:05.048768 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:05.086616 (+ 37848us) log.cc:842] Serialized 52 byte log entry
0806 03:52:05.412557 (+325941us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:05.438875 (+ 26318us) log.cc:842] Serialized 14 byte log entry
0806 03:52:06.279356 (+840481us) spinlock_profiling.cc:243] Waited 6.27 ms on lock 0x7b18003df7e0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d7c75482c 00007f0d79b7f9dc 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9a155 00007f0d7cabd3d0 00007f0d7cacf576 00007f0d7cae00d7
0806 03:52:06.279527 (+ 171us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:06.279739 (+ 212us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:06.279789 (+ 50us) raft_consensus.cc:1671] finished
0806 03:52:06.279989 (+ 200us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:06.294153 (+ 14164us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28850,"prepare.queue_time_us":34673,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":5178,"raft.queue_time_us":31329,"raft.run_cpu_time_us":790,"raft.run_wall_time_us":122122,"spinlock_wait_cycles":14414720,"thread_start_us":325909,"threads_started":2,"wal-append.queue_time_us":378618}
I20260806 03:52:06.263585 5878 consensus_queue.cc:1046] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260806 03:52:06.325750 2372 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
W20260806 03:52:06.327054 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 65660.1us
I20260806 03:52:06.283948 5932 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260806 03:52:06.311985 5942 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:06.342038 2576 pending_rounds.cc:76] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 1
I20260806 03:52:06.342625 2576 pending_rounds.cc:97] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 03:52:06.342978 2576 raft_consensus.cc:2866] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:06.344731 2534 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:06.320576 5922 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:06.329741 5966 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:06.342338 5932 consensus_queue.cc:1225] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 03:52:06.288609 5980 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:06.412451 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 79404.5us
W20260806 03:52:06.413627 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 69133.6us
W20260806 03:52:06.434650 6009 log.cc:924] Time spent T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.093s user 0.001s sys 0.000s
I20260806 03:52:06.439103 5980 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:06.441318 2577 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:52:06.444904 5964 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:06.446274 5992 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:06.447295 5992 consensus_queue.cc:1046] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:06.448273 6016 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:06.449723 5982 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:06.448268 5992 consensus_queue.cc:1225] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260806 03:52:06.489102 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 62130.1us
I20260806 03:52:06.493449 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:06.497251 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 154336us
I20260806 03:52:06.450356 5982 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:06.512725 2436 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:06.512310 5991 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:06.528081 6006 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.227s user 0.002s sys 0.000s
I20260806 03:52:06.520681 5942 raft_consensus.cc:686] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:06.582571 6019 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:06.544929 6033 raft_consensus.cc:3032] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:06.561483 5878 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:52:06.650827 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 152678us
W20260806 03:52:06.668711 2581 consensus_meta.cc:287] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.547s user 0.013s sys 0.002s
I20260806 03:52:06.669474 2581 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:06.678975 2331 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:06.683743 6005 log.cc:924] Time spent T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.299s user 0.000s sys 0.001s
I20260806 03:52:06.704246 5581 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:06.717114 2581 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 03:52:06.602329 6033 raft_consensus.cc:729] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:06.661701 5940 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:06.748770 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:06.750065 2377 raft_consensus.cc:1226] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
W20260806 03:52:06.758023 2331 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.791s (SENT) [suppressed 29 similar messages]
I20260806 03:52:06.759426 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:06.771327 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:06.776062 2375 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:06.777684 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:06.713850 5849 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:06.791792 3542 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:06.793344 2436 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:06.568109 5991 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:06.747339 6019 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:06.723289 5581 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:06.819321 3591 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 389) took 1718 ms (client timeout 1791 ms). Trace:
I20260806 03:52:06.826845 2379 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260806 03:52:06.826977 3591 rpcz_store.cc:269] 0806 03:52:05.101027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:05.117732 (+ 16705us) service_pool.cc:227] Handling call
0806 03:52:06.819149 (+1701417us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b54006cedd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:06.819260 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":408921472}
W20260806 03:52:06.813194 5981 consensus_meta.cc:287] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.560s user 0.016s sys 0.002s
W20260806 03:52:06.842739 6026 log.cc:924] Time spent T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.154s user 0.000s sys 0.001s
I20260806 03:52:06.846076 6036 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:06.847216 5682 consensus_meta.cc:287] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.529s user 0.000s sys 0.011s
I20260806 03:52:06.846918 6036 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:06.847821 5682 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:06.849574 6036 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:06.850381 5924 raft_consensus.cc:2781] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:06.850984 5924 raft_consensus.cc:686] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:06.851991 5924 consensus_queue.cc:237] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:06.816290 5878 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:06.875259 3591 raft_consensus.cc:1226] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:06.761662 5940 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:06.761168 5942 consensus_queue.cc:237] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:06.909417 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 49994.7us
I20260806 03:52:06.872445 5581 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:06.803642 6032 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.096s
I20260806 03:52:06.800477 5849 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:06.909788 5682 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260806 03:52:06.971283 2379 raft_consensus.cc:1226] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:06.973070 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:06.974932 5940 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:06.830888 6033 consensus_queue.cc:260] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:06.992115 6033 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:06.998342 5907 raft_consensus.cc:3032] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:06.999076 5907 raft_consensus.cc:729] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:07.012260 3586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 389) took 1932 ms (client timeout 2148 ms). Trace:
W20260806 03:52:07.013221 3586 rpcz_store.cc:269] 0806 03:52:05.079913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:05.081129 (+ 1216us) service_pool.cc:227] Handling call
0806 03:52:07.011901 (+1930772us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b54005a05d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:07.012146 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1269196032}
I20260806 03:52:07.021737 5991 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:06.971026 6021 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:07.024036 6021 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:06.987388 6032 consensus_queue.cc:1225] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 03:52:06.963759 5982 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:07.041615 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 248582us
I20260806 03:52:07.042948 2436 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.043883 5878 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260806 03:52:07.002082 6034 log.cc:924] Time spent T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.133s user 0.000s sys 0.001s
I20260806 03:52:06.862802 5981 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.053591 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2165) took 1957 ms. Trace:
I20260806 03:52:07.059381 2436 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.079631 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:07.062006 2579 rpcz_store.cc:276] 0806 03:52:05.095802 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:05.096132 (+ 330us) service_pool.cc:227] Handling call
0806 03:52:05.096886 (+ 754us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:05.237138 (+140252us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:05.332705 (+ 95567us) log.cc:842] Serialized 14 byte log entry
0806 03:52:05.589742 (+257037us) log.cc:842] Serialized 14 byte log entry
0806 03:52:05.664938 (+ 75196us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:05.665020 (+ 82us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:05.665085 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:05.955265 (+290180us) spinlock_profiling.cc:243] Waited 6.51 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:06.273417 (+318152us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b1800299c80. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d764 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:06.447598 (+174181us) spinlock_profiling.cc:243] Waited 18.3 ms on lock 0x7b1800299c80. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:07.052852 (+605254us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b1800299c80. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b6af96 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9
0806 03:52:07.053354 (+ 502us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400455bd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:07.053403 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:07.053521 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":156365,"prepare.queue_time_us":97894,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":3105,"raft.queue_time_us":25401,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":12489,"spinlock_wait_cycles":501705728,"thread_start_us":256178,"threads_started":2,"wal-append.queue_time_us":234026,"wal-append.run_cpu_time_us":12085,"wal-append.run_wall_time_us":1449154}
W20260806 03:52:07.068387 5972 consensus_meta.cc:287] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.810s user 0.013s sys 0.005s
W20260806 03:52:07.083379 2576 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s user 0.002s sys 0.001s
W20260806 03:52:07.087401 2576 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s user 0.003s sys 0.001s
I20260806 03:52:07.057056 6021 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:07.063098 5880 raft_consensus.cc:3032] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:07.062300 5981 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:07.074141 5992 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:07.102085 2436 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.106416 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:07.099738 6015 log.cc:924] Time spent T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.750s user 0.004s sys 0.011s
I20260806 03:52:07.100199 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:07.089849 5972 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.103108 5907 consensus_queue.cc:260] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:07.117525 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:07.098228 5880 raft_consensus.cc:729] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:07.105212 5849 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:07.137996 3594 raft_consensus.cc:1226] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:07.143831 2436 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.144589 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:07.146391 2436 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.147680 5878 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:07.148557 5878 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:07.129432 6021 consensus_queue.cc:1046] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.020s
W20260806 03:52:07.149317 5983 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.814s user 0.009s sys 0.007s
I20260806 03:52:07.149575 6021 consensus_queue.cc:1225] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Peer dff94100ffc5485f92cbe6a85addf0b2 log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 03:52:07.149780 5983 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:07.131708 5992 consensus_queue.cc:1046] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260806 03:52:07.153367 5992 consensus_queue.cc:1225] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer dff94100ffc5485f92cbe6a85addf0b2 log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 03:52:07.153923 5983 raft_consensus.cc:3032] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:07.154475 5983 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:07.155792 5983 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:07.121299 5981 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:07.164179 2474 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 03:52:07.126791 6053 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:07.126641 3586 raft_consensus.cc:2362] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8. Re-sending same reply.
W20260806 03:52:07.162019 5888 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.262s user 0.006s sys 0.040s
I20260806 03:52:07.156545 5945 consensus_queue.cc:237] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 3, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.190201 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1937) took 2211 ms. Trace:
I20260806 03:52:07.190954 2331 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:07.190780 2368 rpcz_store.cc:276] 0806 03:52:04.974885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:04.975372 (+ 487us) service_pool.cc:227] Handling call
0806 03:52:07.185214 (+2209842us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65b7c 00007f0d79b6573b 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:52:07.185454 (+ 240us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:07.185673 (+ 219us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:07.185699 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:07.185760 (+ 61us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:07.185789 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:07.185849 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:07.185992 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4398592}
W20260806 03:52:07.177876 5888 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.278s user 0.006s sys 0.041s
I20260806 03:52:07.166528 6019 raft_consensus.cc:686] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
W20260806 03:52:07.185926 5966 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.806s user 0.018s sys 0.000s
I20260806 03:52:07.184930 5983 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:07.170284 5981 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.274832 5981 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:07.211102 5966 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:52:07.279598 2572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2162) took 2228 ms (client timeout 2148 ms). Trace:
I20260806 03:52:07.166638 5849 consensus_queue.cc:1046] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260806 03:52:07.188169 5880 consensus_queue.cc:260] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.172868 5936 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
W20260806 03:52:07.261961 6055 log.cc:924] Time spent T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260806 03:52:07.261927 6019 consensus_queue.cc:237] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:07.264652 5972 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:07.265278 3592 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:07.266428 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:07.245239 6052 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:07.267717 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:07.268864 2332 leader_election.cc:334] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.677s (SENT)
W20260806 03:52:07.281311 2572 rpcz_store.cc:269] 0806 03:52:05.050948 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:05.059242 (+ 8294us) service_pool.cc:227] Handling call
0806 03:52:07.279239 (+2219997us) spinlock_profiling.cc:243] Waited 733 ms on lock 0x7b540040fbd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:07.279445 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1686238080}
W20260806 03:52:07.286808 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 140496us
I20260806 03:52:07.290750 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:07.293216 5961 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 1.535s user 0.006s sys 0.009s
W20260806 03:52:07.295671 6058 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.082s user 0.005s sys 0.004s
I20260806 03:52:07.301954 2479 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:07.304368 2478 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 3.
I20260806 03:52:07.311460 3547 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:07.314234 5913 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:07.319437 5913 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:07.319862 5913 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:07.309028 5880 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:07.306747 5849 consensus_queue.cc:1046] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260806 03:52:07.354175 5849 raft_consensus.cc:3032] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:07.354609 5849 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
W20260806 03:52:07.347632 6061 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260806 03:52:07.359922 6059 log.cc:924] Time spent T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.077s user 0.001s sys 0.001s
I20260806 03:52:07.361467 5966 raft_consensus.cc:3032] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:07.361886 5966 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:07.361969 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2179) took 1040 ms. Trace:
I20260806 03:52:07.362473 2576 rpcz_store.cc:276] 0806 03:52:06.321679 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:06.321834 (+ 155us) service_pool.cc:227] Handling call
0806 03:52:06.341726 (+ 19892us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:06.494378 (+152652us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:06.494410 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:06.524704 (+ 30294us) log.cc:842] Serialized 52 byte log entry
0806 03:52:07.088086 (+563382us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:52:07.088149 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:07.088258 (+ 109us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:07.361738 (+273480us) raft_consensus.cc:1671] finished
0806 03:52:07.361761 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:07.361901 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36012,"raft.queue_time_us":103481,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":146,"spinlock_wait_cycles":53504,"thread_start_us":639953,"threads_started":2,"wal-append.queue_time_us":612487}
I20260806 03:52:07.348765 5794 raft_consensus.cc:3032] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:07.356984 2436 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:07.356730 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 87895.9us
I20260806 03:52:07.308779 6052 raft_consensus.cc:686] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:07.367897 5794 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:07.363013 5966 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.372895 5932 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:07.375023 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:07.378979 2332 leader_election.cc:334] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 4.490s (SENT)
I20260806 03:52:07.355490 5849 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.381417 5907 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:07.395115 2370 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:52:07.404367 2479 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:52:07.405491 2332 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.405941 3547 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:07.406409 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 49512.8us
I20260806 03:52:07.406152 5922 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:07.414776 5922 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:07.420956 2436 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:07.427640 5533 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:07.427846 5878 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:07.428468 5756 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:07.428603 2436 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.429184 5756 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.418109 5474 raft_consensus.cc:2781] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:07.429898 5992 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260806 03:52:07.428805 5878 raft_consensus.cc:686] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:07.431025 5474 raft_consensus.cc:686] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:07.432329 5878 consensus_queue.cc:237] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:07.408175 5972 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260806 03:52:07.390336 6053 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:07.438531 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:07.445765 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:07.446375 2436 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.439363 2483 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:07.450476 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:07.357506 6068 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:07.458034 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:07.468981 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 90052.9us
I20260806 03:52:07.432924 5474 consensus_queue.cc:237] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.428665 5533 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:07.513741 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 73376.7us
I20260806 03:52:07.453909 6053 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:07.465608 5919 raft_consensus.cc:1013] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: attempt to promote peer cf6042e85bef47c4aac9d064d91501d1: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260806 03:52:07.402880 5932 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:52:07.656383 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 239644us
I20260806 03:52:07.641984 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:07.637835 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:07.661509 2574 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:07.707407 2574 raft_consensus.cc:595] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:07.708093 2574 raft_consensus.cc:603] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 03:52:07.678773 2332 leader_election.cc:334] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.764s (SENT)
I20260806 03:52:07.645610 2576 raft_consensus.cc:1261] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:52:07.673568 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:07.637764 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:07.500392 5723 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:07.711630 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 281056us
I20260806 03:52:07.713047 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:07.705452 2584 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:07.718499 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:07.723479 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:07.724023 5926 consensus_queue.cc:1046] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:07.713762 2583 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:07.709108 2332 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.514539 5685 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:07.510710 5533 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:07.509142 5702 raft_consensus.cc:3032] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:07.730837 2579 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260806 03:52:07.711046 2474 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:07.732288 5639 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:07.741824 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:07.656992 6063 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
I20260806 03:52:07.713433 5851 raft_consensus.cc:1013] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: attempt to promote peer cf6042e85bef47c4aac9d064d91501d1: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20260806 03:52:07.657747 6013 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:07.690795 5966 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:07.801474 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 51410.1us
I20260806 03:52:07.760685 2436 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.677486 5849 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:07.656383 5959 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:07.671114 5756 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:07.664898 5888 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:07.690418 6036 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:07.475736 6068 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:07.770726 6033 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.778s user 0.015s sys 0.002s
I20260806 03:52:07.927086 2583 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260806 03:52:07.618834 6052 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.928809 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:07.934674 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 186518us
W20260806 03:52:07.927572 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 177356us
W20260806 03:52:07.931183 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 186753us
I20260806 03:52:07.761547 5702 raft_consensus.cc:729] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:07.937963 5860 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:07.940935 6057 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:07.946597 5913 consensus_meta.cc:287] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.626s user 0.013s sys 0.000s
I20260806 03:52:07.957260 5932 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:07.957695 5880 consensus_meta.cc:287] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.590s user 0.012s sys 0.003s
I20260806 03:52:07.956861 5794 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:07.971763 2583 pending_rounds.cc:76] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 4
I20260806 03:52:07.806696 6053 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:07.990604 2583 pending_rounds.cc:97] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260806 03:52:07.993443 2583 raft_consensus.cc:2866] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:07.806581 5878 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:52:07.990912 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 54151.4us
I20260806 03:52:07.972254 6021 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:07.997085 2436 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:08.001160 2476 raft_consensus.cc:3037] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:07.988641 5983 consensus_meta.cc:287] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.708s user 0.007s sys 0.008s
I20260806 03:52:08.019345 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:07.927258 5959 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.013247 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:08.033878 2577 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260806 03:52:08.027842 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 64218.7us
I20260806 03:52:07.861480 6036 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:08.040541 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:07.807917 6013 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:08.047292 3545 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:08.014603 6033 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:08.045373 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:08.026046 5794 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:07.838578 5888 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:08.058517 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:08.037559 5878 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:08.061770 3545 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.988652 5913 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.059587 2538 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:08.066638 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:07.988909 5860 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.075429 2578 raft_consensus.cc:1261] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:52:08.062711 5933 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:08.045933 5921 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:08.077590 2476 raft_consensus.cc:1261] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 6. (term mismatch)
I20260806 03:52:08.087278 2476 pending_rounds.cc:76] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 5
I20260806 03:52:08.087704 2476 pending_rounds.cc:97] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260806 03:52:08.024202 6021 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:08.123091 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 63602.4us
I20260806 03:52:08.124272 2538 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:08.088871 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:08.090230 2476 raft_consensus.cc:2866] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:08.125149 2577 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:08.061148 5964 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:08.074672 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:08.013151 6053 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:08.122354 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 48102.3us
I20260806 03:52:08.056852 5926 consensus_queue.cc:1035] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
I20260806 03:52:08.134012 2574 raft_consensus.cc:1261] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:52:08.056730 5936 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.066s
I20260806 03:52:08.081054 5886 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:52:08.089859 5932 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:08.089756 5933 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:08.039235 5702 consensus_queue.cc:260] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:08.126668 6043 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:08.128815 6057 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:08.072005 6033 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:08.135227 5915 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:08.155829 2538 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:08.148273 5702 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:08.157644 5915 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:08.165290 6053 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:08.165680 6033 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:08.166208 5755 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:08.166392 6013 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:08.165716 5922 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:08.168637 2538 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:08.201637 5756 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:08.202193 5756 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:08.202656 5756 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:08.208114 6080 consensus_queue.cc:1035] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20260806 03:52:08.210175 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:08.169410 5860 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:08.157673 5435 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:08.213711 2481 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "da90827d7f0646f5bb848777586aae32"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
new_leader_uuid: "cf6042e85bef47c4aac9d064d91501d1"
from {username='slave'} at 127.0.0.1:42512
I20260806 03:52:08.216944 2481 raft_consensus.cc:595] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Received request to transfer leadership to TS cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:08.217294 2481 raft_consensus.cc:603] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:08.149653 5933 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:08.221839 5907 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.815s user 0.011s sys 0.005s
I20260806 03:52:08.223163 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:08.223503 5913 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:08.223963 2476 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:08.126523 5921 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.226110 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:08.226465 5921 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:08.227108 2483 raft_consensus.cc:2396] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:08.190677 6068 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:08.193053 2375 raft_consensus.cc:3032] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:08.194279 5888 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:08.194962 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:08.196175 5915 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:08.240711 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:08.242951 2369 raft_consensus.cc:2396] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 8 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260806 03:52:08.198544 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 47244.2us
I20260806 03:52:08.231243 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:08.237469 2375 raft_consensus.cc:729] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:08.246830 2478 raft_consensus.cc:2396] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260806 03:52:08.165325 6021 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:08.151219 6019 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.095s
I20260806 03:52:08.210609 6036 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:08.249946 6051 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s user 0.024s sys 0.004s
I20260806 03:52:08.264057 3592 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 03:52:08.257720 2375 consensus_queue.cc:260] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.267951 2375 raft_consensus.cc:3037] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:08.154454 6043 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.277444 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:08.262917 5760 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.026s sys 0.066s
W20260806 03:52:08.281464 5760 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s user 0.026s sys 0.067s
I20260806 03:52:08.322412 5932 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:52:08.325630 6080 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:08.326339 6080 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:08.334113 6043 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:08.275496 5959 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:08.335044 6080 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:08.270910 6042 consensus_queue.cc:1035] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
I20260806 03:52:08.339267 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:08.339622 2480 raft_consensus.cc:1261] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:52:08.341296 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:08.341984 2482 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:08.339969 2481 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:52:08.347265 5685 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.594s user 0.012s sys 0.000s
I20260806 03:52:08.348150 5834 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:08.348790 5834 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:08.350375 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 65107.8us
I20260806 03:52:08.361222 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:08.368759 2481 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:52:08.395063 2538 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:08.404635 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:08.405630 2369 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:08.406584 6068 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:08.406318 5756 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:08.410496 3542 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:08.425354 5888 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:08.426098 5888 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.428443 5872 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:08.416229 5533 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:08.429204 5533 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:08.429073 5872 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.429677 5533 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:08.431496 5872 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:08.432710 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:08.441998 2329 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.549s user 0.005s sys 0.019s
W20260806 03:52:08.442497 2329 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.582s user 0.005s sys 0.019s
I20260806 03:52:08.444058 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:08.407136 6068 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:08.419239 5959 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:08.451923 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:08.453979 2379 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
W20260806 03:52:08.453233 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.001s sys 0.026s
I20260806 03:52:08.449981 3547 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
W20260806 03:52:08.467439 2584 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.016s sys 0.024s
W20260806 03:52:08.468089 2584 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.547s user 0.016s sys 0.024s
W20260806 03:52:08.451313 5945 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.005s sys 0.022s
I20260806 03:52:08.469771 5935 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:08.469838 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:08.446038 5682 consensus_queue.cc:1035] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
I20260806 03:52:08.473469 6068 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:08.516287 5915 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:08.454113 5959 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:08.458099 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.688s user 0.001s sys 0.026s
I20260806 03:52:08.480255 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:08.483732 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:08.716504 2574 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.000s sys 0.014s
W20260806 03:52:08.716982 2574 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s user 0.000s sys 0.014s
I20260806 03:52:08.669657 2369 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
W20260806 03:52:08.659505 2479 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.144s user 0.014s sys 0.014s
W20260806 03:52:08.719803 2479 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.204s user 0.014s sys 0.017s
W20260806 03:52:08.484802 5945 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.715s user 0.005s sys 0.023s
I20260806 03:52:08.477137 5932 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:08.667093 3545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.006s sys 0.008s
W20260806 03:52:08.752944 3545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.006s sys 0.009s
W20260806 03:52:08.667814 2583 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.005s sys 0.009s
W20260806 03:52:08.754107 2583 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.006s sys 0.009s
W20260806 03:52:08.666373 2579 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.010s sys 0.044s
I20260806 03:52:08.487030 5888 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:08.675266 2581 raft_consensus.cc:1226] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:08.676528 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:08.453751 5860 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 03:52:08.702231 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 705233us
W20260806 03:52:08.808449 2579 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.772s user 0.016s sys 0.051s
I20260806 03:52:08.674898 5837 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:08.747522 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:08.786360 2537 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:08.751680 6078 log.cc:924] Time spent T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260806 03:52:08.791157 2572 raft_consensus.cc:1226] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260806 03:52:08.706410 5935 raft_consensus.cc:686] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:08.800933 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:08.675849 5894 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:08.809239 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:08.851398 2369 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:08.718933 5959 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:08.705590 2376 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:52:08.825919 2372 raft_consensus.cc:2396] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 03:52:08.807396 5860 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:08.836838 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 82636us
W20260806 03:52:08.843240 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 1.16442e+06us
I20260806 03:52:08.864121 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:08.704938 5933 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.827208 2436 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:08.864511 2332 leader_election.cc:334] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.773s (SENT)
I20260806 03:52:08.831419 5756 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:08.851621 5888 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:08.866807 5760 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.010s sys 0.019s
I20260806 03:52:08.876384 5945 consensus_queue.cc:1046] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
W20260806 03:52:08.891434 5760 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.569s user 0.010s sys 0.019s
W20260806 03:52:08.891609 2438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.007s sys 0.019s
W20260806 03:52:08.888429 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 61433.9us
W20260806 03:52:08.892153 2438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.899s user 0.008s sys 0.019s
I20260806 03:52:08.794013 5932 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:08.896258 6083 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260806 03:52:08.897114 5933 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:08.891274 6082 log.cc:924] Time spent T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.138s user 0.001s sys 0.001s
W20260806 03:52:08.897725 2577 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.008s sys 0.024s
W20260806 03:52:08.901461 2577 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s user 0.008s sys 0.024s
I20260806 03:52:08.869115 5860 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:08.898226 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:08.892143 5945 raft_consensus.cc:3032] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 03:52:08.902233 2477 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s user 0.009s sys 0.030s
I20260806 03:52:08.920117 2436 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:08.921170 2477 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s user 0.009s sys 0.030s
W20260806 03:52:08.921383 2476 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.003s sys 0.035s
W20260806 03:52:08.942075 2476 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.538s user 0.003s sys 0.035s
I20260806 03:52:08.942209 5794 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:08.838569 5837 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.889243 6090 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:08.902333 6068 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:08.998924 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:08.945143 6052 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.766s user 0.013s sys 0.034s
I20260806 03:52:08.902024 6021 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:09.009747 2576 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:08.920185 6089 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:08.943013 5794 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:08.920827 5945 raft_consensus.cc:729] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
W20260806 03:52:09.010661 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 526842us
W20260806 03:52:09.013157 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 340889us
I20260806 03:52:08.863308 5894 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:08.995268 6090 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:09.006703 5935 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.022552 6021 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.008184 6068 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:52:09.005769 6052 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s user 0.013s sys 0.035s
I20260806 03:52:09.007920 5581 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:09.034783 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 238) took 1064 ms. Trace:
I20260806 03:52:08.944945 5760 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:09.051651 2583 rpcz_store.cc:276] 0806 03:52:07.970801 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:07.970985 (+ 184us) service_pool.cc:227] Handling call
0806 03:52:07.971476 (+ 491us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:08.068338 (+ 96862us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:08.068369 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:08.068950 (+ 581us) log.cc:842] Serialized 52 byte log entry
0806 03:52:08.754629 (+685679us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:08.830601 (+ 75972us) log.cc:842] Serialized 14 byte log entry
0806 03:52:08.837999 (+ 7398us) log.cc:842] Serialized 14 byte log entry
0806 03:52:08.838234 (+ 235us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:08.838325 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:09.034454 (+196129us) raft_consensus.cc:1671] finished
0806 03:52:09.034486 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:09.034693 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":95808,"prepare.queue_time_us":215399,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":108,"raft.queue_time_us":113023,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":16297,"spinlock_wait_cycles":34411008,"thread_start_us":628362,"threads_started":1,"wal-append.queue_time_us":911937}
I20260806 03:52:09.042752 5959 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.022481 5915 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:09.064522 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:09.043987 6090 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:09.065430 5760 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:09.081866 5794 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:09.084789 5916 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260806 03:52:09.085700 6021 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:09.086704 5834 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:09.088960 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2191) took 1405 ms. Trace:
I20260806 03:52:09.111362 2578 rpcz_store.cc:276] 0806 03:52:07.683226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:07.683411 (+ 185us) service_pool.cc:227] Handling call
0806 03:52:07.683860 (+ 449us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:08.081085 (+397225us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:09.088615 (+1007530us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540028d7d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:09.088871 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":890794880}
I20260806 03:52:09.096426 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:09.099742 5933 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:09.106097 6093 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:09.023079 6089 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:09.117539 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 246799us
W20260806 03:52:09.127269 2332 leader_election.cc:334] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.068s (SENT)
I20260806 03:52:09.129842 2332 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:09.062047 5581 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.104199 5894 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:09.120572 6021 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:09.132241 6084 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:09.137686 6036 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:52:09.181492 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 54254.7us
W20260806 03:52:09.193874 2332 leader_election.cc:334] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.213s (SENT)
I20260806 03:52:09.093310 5945 consensus_queue.cc:260] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:09.129153 5916 consensus_queue.cc:1035] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.042s
I20260806 03:52:09.123011 5933 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.125501 6093 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.224429 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:09.173873 6021 raft_consensus.cc:686] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:09.223269 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1115) took 1850 ms. Trace:
I20260806 03:52:09.233873 2373 rpcz_store.cc:276] 0806 03:52:07.372637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:07.373446 (+ 809us) service_pool.cc:227] Handling call
0806 03:52:07.373853 (+ 407us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:07.745453 (+371600us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:07.745498 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:07.745556 (+ 58us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:07.745602 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:09.223019 (+1477417us) spinlock_profiling.cc:243] Waited 371 ms on lock 0x7b5400012cd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:09.223075 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:09.223197 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":853929856}
W20260806 03:52:09.232874 2332 leader_election.cc:334] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.218s (SENT)
I20260806 03:52:09.236424 2332 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:09.267573 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 365004us
I20260806 03:52:09.268764 2436 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:09.269167 5581 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:09.270526 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2205) took 1041 ms. Trace:
I20260806 03:52:09.270991 2577 rpcz_store.cc:276] 0806 03:52:08.229563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:08.229945 (+ 382us) service_pool.cc:227] Handling call
0806 03:52:08.230385 (+ 440us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:08.251876 (+ 21491us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:08.251912 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:08.262437 (+ 10525us) log.cc:842] Serialized 52 byte log entry
0806 03:52:08.901995 (+639558us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:52:08.902043 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:08.902126 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:09.270236 (+368110us) raft_consensus.cc:1671] finished
0806 03:52:09.270261 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:09.270447 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14916,"raft.queue_time_us":13687,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":125,"thread_start_us":557581,"threads_started":1,"wal-append.queue_time_us":863109}
I20260806 03:52:09.301455 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1910) took 1868 ms. Trace:
I20260806 03:52:09.305802 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:09.311072 2478 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:09.314394 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:09.316537 2477 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:09.312654 2479 rpcz_store.cc:276] 0806 03:52:07.432633 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:07.459141 (+ 26508us) service_pool.cc:227] Handling call
0806 03:52:07.459551 (+ 410us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:07.459861 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 5
0806 03:52:08.720375 (+1260514us) log.cc:842] Serialized 14 byte log entry
0806 03:52:09.300989 (+580614us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:09.301060 (+ 71us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:09.301119 (+ 59us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:09.301201 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:09.301369 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41792,"prepare.queue_time_us":473216,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"thread_start_us":1751566,"threads_started":3,"wal-append.queue_time_us":405440}
I20260806 03:52:09.318212 2482 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:09.221210 6068 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:09.220672 5932 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:09.296517 5533 consensus_meta.cc:287] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.866s user 0.011s sys 0.007s
I20260806 03:52:09.354892 2375 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
W20260806 03:52:09.372452 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 139630us
W20260806 03:52:09.376475 2332 connection.cc:232] RPC call timeout handler was delayed by -1.5373829039745033s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
W20260806 03:52:09.376961 2332 connection.cc:232] RPC call timeout handler was delayed by -1.381682903971523s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1785s for a response
I20260806 03:52:09.260296 5837 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:09.380507 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:09.381237 2370 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:09.382452 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:09.383317 2372 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
W20260806 03:52:09.386446 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 74202.3us
W20260806 03:52:09.387959 2481 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.014s sys 0.020s
W20260806 03:52:09.388361 2481 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s user 0.014s sys 0.020s
I20260806 03:52:09.297297 6051 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:09.315207 6086 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:09.315140 5834 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:09.190034 6084 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.412516 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 276) took 1040 ms. Trace:
I20260806 03:52:09.424666 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:09.429006 2480 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:09.431653 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:09.427965 2481 rpcz_store.cc:276] 0806 03:52:08.371753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:08.374584 (+ 2831us) service_pool.cc:227] Handling call
0806 03:52:08.374976 (+ 392us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:08.375245 (+ 269us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:08.791213 (+415968us) log.cc:842] Serialized 14 byte log entry
0806 03:52:09.412066 (+620853us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:09.412129 (+ 63us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:09.412182 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:09.412264 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:09.412435 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44008,"prepare.queue_time_us":626852,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"thread_start_us":973031,"threads_started":2}
I20260806 03:52:09.437934 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:09.443226 3594 raft_consensus.cc:2396] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 03:52:09.444097 3593 raft_consensus.cc:1261] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 03:52:09.462220 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:52:09.425628 6107 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.358s user 0.002s sys 0.000s
I20260806 03:52:09.414180 5932 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:09.410898 5959 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:09.378852 6036 consensus_queue.cc:1046] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20260806 03:52:09.423491 6068 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:09.477005 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:09.488359 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 106345us
I20260806 03:52:09.389425 5533 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:09.481251 2579 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s user 0.011s sys 0.022s
I20260806 03:52:09.464633 6021 consensus_queue.cc:237] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:09.440402 5834 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.434753 6086 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:09.502686 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 120964us
I20260806 03:52:09.644829 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:09.537639 2575 raft_consensus.cc:1226] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
W20260806 03:52:09.535425 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.008s sys 0.003s
I20260806 03:52:09.495550 3590 raft_consensus.cc:2379] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:09.496708 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:09.648803 3545 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:09.515075 2370 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:09.494122 3542 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:09.647792 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.010s sys 0.009s
I20260806 03:52:09.651494 3594 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
W20260806 03:52:09.644169 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 375524us
I20260806 03:52:09.646312 2477 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:09.538985 3592 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36468 (request call id 408) took 1146 ms. Trace:
W20260806 03:52:09.685598 2579 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s user 0.014s sys 0.030s
I20260806 03:52:09.567517 2235 catalog_manager.cc:5495] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 4, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 4 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 03:52:09.489111 6115 log.cc:924] Time spent T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.366s user 0.002s sys 0.000s
W20260806 03:52:09.581661 6138 log.cc:924] Time spent T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
W20260806 03:52:09.566326 6099 log.cc:924] Time spent T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.377s user 0.000s sys 0.001s
W20260806 03:52:09.575549 6133 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
W20260806 03:52:09.580646 6131 log.cc:924] Time spent T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20260806 03:52:09.576217 6125 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.406s user 0.000s sys 0.002s
W20260806 03:52:09.760923 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 266914us
I20260806 03:52:09.480649 5916 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:09.532323 6096 log.cc:924] Time spent T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.367s user 0.001s sys 0.000s
W20260806 03:52:09.762805 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 108473us
I20260806 03:52:09.707423 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:09.685966 3592 rpcz_store.cc:276] 0806 03:52:08.392733 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:08.392911 (+ 178us) service_pool.cc:227] Handling call
0806 03:52:08.394722 (+ 1811us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:08.395233 (+ 511us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:08.395270 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:08.407932 (+ 12662us) log.cc:842] Serialized 548 byte log entry
0806 03:52:08.852386 (+444454us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:08.852432 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:08.852513 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:09.538687 (+686174us) raft_consensus.cc:1671] finished
0806 03:52:09.538713 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:09.538908 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":444042,"threads_started":1,"wal-append.queue_time_us":625331}
W20260806 03:52:09.620182 6148 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260806 03:52:09.794143 2436 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:09.791524 2332 connection.cc:232] RPC call timeout handler was delayed by -1.3267829038668424s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18460000000000001s for a response
I20260806 03:52:09.492053 6143 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:09.583725 5533 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:09.603960 6036 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:09.560109 3586 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:09.539330 6051 raft_consensus.cc:3032] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:09.798765 2534 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:09.488514 6147 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:09.678274 5860 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.706s user 0.016s sys 0.000s
I20260806 03:52:09.807890 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:09.486687 5760 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:09.814406 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:09.521291 6089 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:09.567281 6135 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:09.577164 6093 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:09.805827 5888 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:09.486750 5932 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:09.929275 2537 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:09.937714 2332 connection.cc:232] RPC call timeout handler was delayed by -1.2862829039804637s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18910000000000002s for a response
W20260806 03:52:09.972882 2332 connection.cc:232] RPC call timeout handler was delayed by -1.2619829040486366s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
W20260806 03:52:09.973412 2332 connection.cc:232] RPC call timeout handler was delayed by -1.1341829039156437s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20600000000000002s for a response
W20260806 03:52:09.973433 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 44312.6us
W20260806 03:52:09.973958 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0630829038564116s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21389999999999998s for a response
I20260806 03:52:09.652014 5913 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 03:52:09.940828 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 142211us
I20260806 03:52:09.506131 6068 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:09.936952 6051 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:09.845219 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2197) took 2113 ms. Trace:
I20260806 03:52:09.895533 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:09.901734 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 80895.3us
I20260806 03:52:09.953492 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:09.950371 2368 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:52:09.983676 2332 leader_election.cc:334] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.593s (SENT)
I20260806 03:52:09.948688 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 292) took 1266 ms. Trace:
I20260806 03:52:10.098075 2332 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:09.827127 6139 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.175s user 0.000s sys 0.000s
I20260806 03:52:09.804610 5965 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:10.097947 2374 rpcz_store.cc:276] 0806 03:52:08.682808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:08.682971 (+ 163us) service_pool.cc:227] Handling call
0806 03:52:08.683276 (+ 305us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:08.683502 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:09.051489 (+367987us) log.cc:842] Serialized 14 byte log entry
0806 03:52:09.948186 (+896697us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:09.948264 (+ 78us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:09.948333 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:09.948442 (+ 109us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:09.948613 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8756,"prepare.queue_time_us":214383,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":75196,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":24751,"thread_start_us":1195827,"threads_started":3,"wal-append.queue_time_us":512978}
W20260806 03:52:09.837864 6081 log.cc:924] Time spent T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 1.012s user 0.004s sys 0.003s
I20260806 03:52:09.652675 6084 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:09.834004 5834 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:09.815677 6136 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:10.008728 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:10.037463 2478 consensus_meta.cc:287] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.700s user 0.013s sys 0.001s
I20260806 03:52:10.007736 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:10.137295 2478 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:10.137564 2578 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:10.100142 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 116522us
I20260806 03:52:09.942035 2377 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:52:10.139308 2332 leader_election.cc:334] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.606s (SENT)
I20260806 03:52:10.140374 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.010182 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.142247 2583 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:52:10.105651 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 122136us
I20260806 03:52:09.962803 5760 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:09.999835 6093 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:09.955899 5916 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.098901 2584 rpcz_store.cc:276] 0806 03:52:07.731455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:07.731827 (+ 372us) service_pool.cc:227] Handling call
0806 03:52:07.732175 (+ 348us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:07.733866 (+ 1691us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:52:08.483506 (+749640us) log.cc:842] Serialized 14 byte log entry
0806 03:52:09.844275 (+1360769us) spinlock_profiling.cc:243] Waited 2.81 ms on lock 0x7b18001ae420. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b144ca 00007f0d831f5f99 00007f0d799d8990 00007f0d79b9a864 00007f0d79b9a155 00007f0d8049a167 00007f0d7cacf807 00007f0d7cadef0f 00007f0d7cadee88
0806 03:52:09.844556 (+ 281us) log.cc:842] Serialized 14 byte log entry
0806 03:52:09.844760 (+ 204us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:09.844811 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:09.844864 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:09.844948 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:09.845111 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20669,"prepare.queue_time_us":1407633,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":105,"raft.queue_time_us":21918,"spinlock_wait_cycles":12670976,"thread_start_us":1000265,"threads_started":2,"wal-append.queue_time_us":593697}
I20260806 03:52:10.109339 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:09.945614 5933 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:10.116417 3547 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.667s user 0.020s sys 0.048s
W20260806 03:52:10.162871 3547 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.714s user 0.021s sys 0.049s
W20260806 03:52:10.163321 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 714231us
I20260806 03:52:10.166105 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.167799 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.168490 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.158497 3542 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:10.117869 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:09.980751 6097 log.cc:924] Time spent T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
W20260806 03:52:10.133040 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.026s sys 0.110s
I20260806 03:52:10.144343 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:10.173810 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.620s user 0.026s sys 0.111s
I20260806 03:52:10.141413 2368 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:10.013962 5702 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:10.013293 6147 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:09.948881 6036 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.149899 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.193672 2373 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:09.997452 5932 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:09.995512 6089 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:10.151314 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:09.989629 6165 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.107996 5965 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:10.014070 6161 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:10.169387 2477 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:10.170461 2374 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:09.951432 5860 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:09.825142 6086 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.198372 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.227320 2368 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:10.198776 5894 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:10.162128 6111 log.cc:924] Time spent T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.967s user 0.009s sys 0.009s
I20260806 03:52:10.109193 5888 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.210007 2377 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:10.145221 6169 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:10.215066 2584 raft_consensus.cc:1226] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260806 03:52:10.216219 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:09.979694 6135 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.219956 5965 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:10.229974 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.198645 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.230556 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:10.236140 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 294964us
I20260806 03:52:10.168746 2583 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
W20260806 03:52:10.230966 6090 consensus_meta.cc:287] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.116s user 0.016s sys 0.001s
I20260806 03:52:10.300422 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.140924 6126 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.077s
I20260806 03:52:10.207257 5702 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.026s
I20260806 03:52:10.208318 6170 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:10.233681 6084 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.157851 5760 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.255554 2374 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
W20260806 03:52:10.250964 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 563783us
I20260806 03:52:10.357448 2436 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:10.219591 6089 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.301955 2377 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:44596
I20260806 03:52:10.180433 6093 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.145310 6136 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:09.953550 6143 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:10.354300 6104 log.cc:924] Time spent T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.105s user 0.000s sys 0.001s
I20260806 03:52:10.359061 2436 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:10.301357 2368 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:10.250861 6021 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:10.364511 2377 raft_consensus.cc:595] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:10.380493 2377 raft_consensus.cc:603] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 03:52:10.354606 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 213958us
I20260806 03:52:10.303117 5894 consensus_queue.cc:1046] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.063s
W20260806 03:52:10.382398 2332 leader_election.cc:334] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.638s (SENT)
W20260806 03:52:10.383615 2332 leader_election.cc:334] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.642s (SENT)
I20260806 03:52:10.340629 5932 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:10.376210 2373 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:10.190814 6051 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.354368 6180 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:10.384043 2332 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:10.220144 6165 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:10.340121 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 166687us
I20260806 03:52:10.359398 6170 raft_consensus.cc:686] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:10.390372 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 277) took 2018 ms. Trace:
I20260806 03:52:10.207736 6147 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:10.341195 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 156634us
I20260806 03:52:10.401291 2483 rpcz_store.cc:276] 0806 03:52:08.372177 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:08.372430 (+ 253us) service_pool.cc:227] Handling call
0806 03:52:08.372902 (+ 472us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:08.373394 (+ 492us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:08.373440 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:08.374074 (+ 634us) log.cc:842] Serialized 26 byte log entry
0806 03:52:08.811328 (+437254us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:08.811379 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:08.811482 (+ 103us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:10.014511 (+1203029us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:10.380370 (+365859us) raft_consensus.cc:1671] finished
0806 03:52:10.380399 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:10.390261 (+ 9862us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":8162,"spinlock_wait_cycles":5614464,"thread_start_us":392331,"threads_started":1,"wal-append.queue_time_us":712218}
I20260806 03:52:10.228821 6161 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:10.417773 2332 leader_election.cc:334] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.785s (SENT)
I20260806 03:52:10.418489 2332 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:10.340461 6090 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.376528 6068 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:10.430510 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 92096.3us
I20260806 03:52:10.363713 5755 consensus_queue.cc:1046] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260806 03:52:10.416422 5723 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:10.451074 2478 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:10.362118 6084 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.459225 5888 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:10.439342 5933 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:10.459990 5888 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.460644 5888 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.461164 6084 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:10.451864 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 92901.3us
I20260806 03:52:10.466902 2436 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:10.407454 6180 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:10.454697 2332 leader_election.cc:334] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.846s (SENT)
I20260806 03:52:10.431474 5860 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:10.473466 2332 leader_election.cc:334] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.918s (SENT)
W20260806 03:52:10.477203 6177 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260806 03:52:10.462884 5888 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:10.481118 2538 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:10.463229 6147 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:10.488765 6174 log.cc:924] Time spent T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.272s user 0.001s sys 0.000s
I20260806 03:52:10.491286 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2210) took 1817 ms. Trace:
W20260806 03:52:10.584879 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 111438us
I20260806 03:52:10.584724 2575 rpcz_store.cc:276] 0806 03:52:08.673544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:08.673779 (+ 235us) service_pool.cc:227] Handling call
0806 03:52:08.713673 (+ 39894us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:09.646024 (+932351us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:10.490895 (+844871us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b54004466d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:10.491194 (+ 299us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1894374272}
I20260806 03:52:10.494185 6135 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.497908 6143 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.499866 6161 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.501745 5760 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:10.495730 5963 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:52:10.603379 6175 log.cc:924] Time spent T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.177s user 0.001s sys 0.000s
I20260806 03:52:10.602969 5963 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.516621 6093 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.606145 5916 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.517736 6036 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.519663 6089 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.607745 6089 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260806 03:52:10.607933 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 141162us
I20260806 03:52:10.609484 2436 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:10.610641 6161 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:10.520196 5886 raft_consensus.cc:3032] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:10.445392 6068 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:10.430388 6183 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:10.528971 2578 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
W20260806 03:52:10.604449 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 125303us
I20260806 03:52:10.514060 6136 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.605276 5888 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:52:10.615068 2332 leader_election.cc:334] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.163s (SENT)
W20260806 03:52:10.415735 6086 consensus_queue.cc:1159] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cf6042e85bef47c4aac9d064d91501d1" responder_term: 6 status { last_received { term: 6 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 5 } }
I20260806 03:52:10.636883 2538 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:10.461175 6165 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:10.619261 5886 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
W20260806 03:52:10.644220 2332 leader_election.cc:334] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.228s (SENT)
W20260806 03:52:10.645298 2332 leader_election.cc:334] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.637s (SENT)
I20260806 03:52:10.645721 2332 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:10.590281 6021 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:10.647274 2538 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:10.648600 6180 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:10.642120 5888 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:10.649044 5032 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:10.649324 5888 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:10.650012 5032 raft_consensus.cc:686] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:10.470657 6170 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.653811 5963 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.655491 5945 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:10.671058 3542 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:10.666204 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.673516 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:10.675529 2483 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:10.677697 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.678324 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:10.677712 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 416998us
I20260806 03:52:10.678937 2482 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:10.693205 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.694416 2484 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:52:10.695644 2332 leader_election.cc:334] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.965s (SENT)
W20260806 03:52:10.696132 2579 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.752s user 0.007s sys 0.008s
I20260806 03:52:10.696211 2332 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:10.696564 2579 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.753s user 0.007s sys 0.008s
I20260806 03:52:10.681201 2480 raft_consensus.cc:1226] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:10.680145 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:10.682288 2429 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:10.700568 2332 leader_election.cc:334] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 3.128s (SENT)
I20260806 03:52:10.701112 2332 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:10.722656 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.724283 2484 raft_consensus.cc:2396] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 03:52:10.721210 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.726226 2480 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:10.674255 5794 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:10.678057 6201 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:10.651463 5032 consensus_queue.cc:237] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:10.732000 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:10.740005 2332 leader_election.cc:334] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 4.974s (SENT)
I20260806 03:52:10.743261 2476 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:10.693817 6192 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:10.679620 6181 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:10.742214 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:10.699692 5916 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:10.726611 6087 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:10.727283 5760 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:10.755710 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:10.792265 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 70503.9us
I20260806 03:52:10.660351 6021 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.793221 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.794013 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.726817 6180 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:10.730060 5933 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:10.799582 2370 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:10.791092 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 03:52:10.820993 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:10.821643 6189 log.cc:924] Time spent T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260806 03:52:10.748116 6016 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.743040 5932 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.748975 6201 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.828598 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:10.835947 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:10.844290 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.742s user 0.009s sys 0.032s
W20260806 03:52:10.844821 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.743s user 0.009s sys 0.033s
W20260806 03:52:10.845176 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 743976us
I20260806 03:52:10.847440 2538 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:10.825918 6181 raft_consensus.cc:686] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:10.823814 5933 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:10.729343 6185 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:10.824942 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:10.871626 2476 raft_consensus.cc:2354] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6.
I20260806 03:52:10.872681 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:10.873333 2474 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:10.731045 5533 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.826609 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2198) took 2803 ms. Trace:
I20260806 03:52:10.830056 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.878686 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:10.879179 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 178465us
I20260806 03:52:10.873785 3542 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:10.837695 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 97733.2us
I20260806 03:52:10.880121 2429 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:10.748591 6192 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:10.857640 2538 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:10.864002 2534 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:10.866338 2370 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:10.870700 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:10.874269 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.875494 6164 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:10.875849 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.877620 2579 rpcz_store.cc:276] 0806 03:52:08.023658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:08.023838 (+ 180us) service_pool.cc:227] Handling call
0806 03:52:08.024297 (+ 459us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:08.024608 (+ 311us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:08.812906 (+788298us) log.cc:842] Serialized 14 byte log entry
0806 03:52:10.702166 (+1889260us) log.cc:842] Serialized 14 byte log entry
0806 03:52:10.826033 (+123867us) log.cc:842] Serialized 14 byte log entry
0806 03:52:10.826229 (+ 196us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:10.826277 (+ 48us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:10.826334 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:10.826409 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:10.826538 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":722232,"prepare.queue_time_us":213966,"prepare.run_cpu_time_us":177,"prepare.run_wall_time_us":6602,"raft.queue_time_us":616047,"raft.run_cpu_time_us":706,"raft.run_wall_time_us":351338,"thread_start_us":2404846,"threads_started":3,"wal-append.queue_time_us":1166932}
I20260806 03:52:10.741322 6209 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.881592 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.888257 2477 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:10.894888 2474 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:10.902940 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.746718 6090 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:10.907819 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.913254 2484 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:10.908285 2429 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:10.869210 5933 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:10.912189 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:10.912326 2534 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:10.914963 2429 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:10.817868 5916 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.909741 2478 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:10.924054 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.918931 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:10.869588 6185 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:10.927063 2534 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:10.877041 5837 consensus_queue.cc:1035] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260806 03:52:10.884140 6192 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:10.931715 2579 raft_consensus.cc:1203] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Deduplicated request from leader. Original: 3.2->[6.3-6.3] Dedup: 6.3->[]
I20260806 03:52:10.946655 2579 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260806 03:52:10.932643 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.948442 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.949365 2478 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
W20260806 03:52:10.954643 2332 leader_election.cc:334] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.612s (SENT)
I20260806 03:52:10.829866 6180 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.932883 6205 consensus_queue.cc:1046] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
W20260806 03:52:10.933890 6156 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260806 03:52:10.934203 5685 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:10.937727 2235 catalog_manager.cc:5495] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:10.939155 6201 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:52:10.940200 6121 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:10.962292 6180 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:10.962622 6121 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:10.963088 6121 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:10.872071 6016 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:10.896121 5755 consensus_queue.cc:1225] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer cf6042e85bef47c4aac9d064d91501d1 log is divergent from this leader: its last log entry 4.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260806 03:52:10.976835 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.977661 2373 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:10.978685 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.979190 2429 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:10.979570 2373 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:10.943663 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.986200 2374 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:10.986887 2429 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:10.987483 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.944108 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.944684 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.945437 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.956252 2538 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:10.957297 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.958607 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260806 03:52:10.958947 6202 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:10.991169 2377 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:10.959604 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.906558 5833 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:10.989351 2372 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:10.959414 6154 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260806 03:52:10.898097 5860 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:11.004916 2584 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:11.009016 2374 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:10.943914 6068 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:10.942694 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.983156 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.984850 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.976547 6180 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.996047 2369 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:52:10.997905 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 70922.6us
I20260806 03:52:10.999365 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.918697 6013 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:10.869529 5932 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.011857 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.988927 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:10.975114 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:10.982434 6052 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:10.941905 6183 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:10.946858 6185 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:11.021534 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:10.995777 6090 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:11.158190 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 244) took 2294 ms. Trace:
I20260806 03:52:11.166882 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2009) took 1466 ms. Trace:
I20260806 03:52:11.177206 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:10.941735 6136 consensus_queue.cc:1046] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 3, Time since last communication: 0.083s
I20260806 03:52:11.156620 6163 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:10.897404 5533 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.020308 6161 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:11.057485 2478 raft_consensus.cc:2354] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
W20260806 03:52:11.180336 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 224173us
I20260806 03:52:11.052188 6068 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:11.178196 2581 rpcz_store.cc:276] 0806 03:52:08.863950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:08.865547 (+ 1597us) service_pool.cc:227] Handling call
0806 03:52:08.865910 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:08.919635 (+ 53725us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:08.919670 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:08.943903 (+ 24233us) log.cc:842] Serialized 26 byte log entry
0806 03:52:09.302256 (+358353us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:09.302297 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:11.059375 (+1757078us) spinlock_profiling.cc:243] Waited 51.4 ms on lock 0x7b540040f6d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:11.059445 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:11.059517 (+ 72us) raft_consensus.cc:1671] finished
0806 03:52:11.059531 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:11.157961 (+ 98430us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1663,"raft.queue_time_us":2502,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":20854,"spinlock_wait_cycles":118299008,"thread_start_us":290711,"threads_started":1,"wal-append.queue_time_us":506869}
I20260806 03:52:11.278450 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:11.277509 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 226283us
I20260806 03:52:11.178340 6180 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.290207 6028 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:52:11.292734 6171 sasl_common.cc:323] Time spent creating new SASL server: real 0.294s user 0.000s sys 0.001s
I20260806 03:52:11.019770 6202 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:11.297693 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.159437 2575 raft_consensus.cc:2379] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:11.158595 6183 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:11.178895 2379 rpcz_store.cc:276] 0806 03:52:09.700724 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:09.700961 (+ 237us) service_pool.cc:227] Handling call
0806 03:52:09.814597 (+113636us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:10.094372 (+279775us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:10.094453 (+ 81us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:10.094517 (+ 64us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:10.094563 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:11.166490 (+1071927us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b540014f854. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:11.166563 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:11.166791 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":463970688}
I20260806 03:52:11.305239 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.341161 2480 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:11.179961 6090 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:11.312161 2534 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:11.316051 2373 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:11.233366 6167 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:11.078320 6181 consensus_queue.cc:237] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:10.921442 6209 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.340921 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.178296 6129 consensus_queue.cc:1046] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.308s
I20260806 03:52:11.283232 6163 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:11.354514 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.350438 2584 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:11.367233 2534 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:11.321388 6068 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:11.373720 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.375067 2375 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:52:11.178571 6052 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.022694 5860 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.349506 6167 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:11.388216 6167 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:11.362177 5829 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:11.349745 6090 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:11.343474 6202 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:11.401674 2483 consensus_meta.cc:287] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.725s user 0.016s sys 0.000s
I20260806 03:52:11.391228 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.389580 2578 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:11.402838 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.352623 6028 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:11.343204 6183 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:11.408898 2483 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:11.372526 6163 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:11.410023 2479 raft_consensus.cc:1226] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:11.413584 2370 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:11.417582 2538 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:11.421149 2480 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:11.303790 6203 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.088s
I20260806 03:52:11.425760 2584 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:11.431461 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.432248 2480 raft_consensus.cc:2354] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4.
W20260806 03:52:11.433091 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 65969.9us
I20260806 03:52:11.435531 2534 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:11.052713 5833 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:11.442203 6013 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:52:11.443389 6204 log.cc:924] Time spent T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.262s user 0.000s sys 0.001s
I20260806 03:52:11.401268 6021 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:11.447294 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.456655 2584 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:11.408597 6028 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:11.452929 6051 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:11.422621 5972 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
W20260806 03:52:11.443389 6215 log.cc:924] Time spent T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20260806 03:52:11.468528 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.476925 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.480957 6201 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:11.482323 2574 raft_consensus.cc:2396] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:52:11.493280 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.466941 6164 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:11.495486 2481 raft_consensus.cc:2354] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7.
I20260806 03:52:11.451006 6013 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:11.428974 6154 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260806 03:52:11.471025 6120 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:11.505026 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.509678 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.520956 6161 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:11.521759 6161 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:11.528167 6223 log.cc:924] Time spent T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260806 03:52:11.530417 6209 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:11.533635 6052 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:11.481876 6201 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:11.521842 5860 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:11.550858 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.551882 2484 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:11.548491 2429 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:11.550285 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:11.551196 2332 leader_election.cc:334] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.741s (SENT)
I20260806 03:52:11.555598 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.499469 5916 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:11.559402 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:11.552865 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:11.505522 6013 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:11.563539 2429 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:11.560444 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 54847.3us
I20260806 03:52:11.565953 2429 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:11.567142 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.506032 6136 consensus_queue.cc:1225] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer dff94100ffc5485f92cbe6a85addf0b2 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 03:52:11.568778 6021 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:11.573665 6180 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:11.546660 6086 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:11.575398 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.614547 2476 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:11.576504 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.617926 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:11.618772 2369 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:11.577631 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.579335 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.582322 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:11.597131 6222 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.107s user 0.001s sys 0.000s
I20260806 03:52:11.597764 2575 raft_consensus.cc:3032] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:11.634140 2575 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:11.598400 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.597219 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:11.599354 6221 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20260806 03:52:11.600346 6016 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:11.507988 6164 raft_consensus.cc:686] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:11.610028 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.621696 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.685195 2484 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:11.624830 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:11.622081 6028 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:11.605664 6021 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:11.631136 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.636786 2575 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.615235 6136 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:11.693044 6164 consensus_queue.cc:237] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:11.569183 5932 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:11.696661 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.686828 5533 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:11.706812 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.633073 6180 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:11.708200 6028 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:11.715416 2377 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:11.716207 6021 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:11.758242 2332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.047s (SENT) [suppressed 29 similar messages]
I20260806 03:52:11.760958 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.720541 5886 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.724470 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:11.688318 5965 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.344s user 0.007s sys 0.012s
I20260806 03:52:11.743052 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:11.723781 6136 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:11.748582 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:11.745893 6180 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:11.751543 5933 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.804s user 0.013s sys 0.001s
I20260806 03:52:11.752252 2379 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
W20260806 03:52:11.752666 5888 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.103s user 0.015s sys 0.007s
W20260806 03:52:11.753976 5945 consensus_meta.cc:287] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.094s user 0.011s sys 0.010s
I20260806 03:52:11.633951 6086 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:11.784534 2332 leader_election.cc:334] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.791s (SENT)
I20260806 03:52:11.813441 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.844098 6227 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:11.841588 2476 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:11.864944 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:11.850816 5932 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:11.882792 6084 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:11.860175 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:11.633963 6183 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:11.904441 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:11.913009 2477 raft_consensus.cc:2362] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5. Re-sending same reply.
I20260806 03:52:11.904546 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:11.914160 2370 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:11.914469 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 130026us
I20260806 03:52:11.817960 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.884121 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.881295 2481 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:11.908490 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:11.901088 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 335234us
I20260806 03:52:11.919238 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.917940 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:11.917181 2332 leader_election.cc:334] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.841s (SENT)
I20260806 03:52:11.901762 3590 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:11.884907 2583 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:52:11.895211 2474 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:11.924414 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.924404 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.926246 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:11.926842 2574 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:52:11.927390 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:11.925218 2474 raft_consensus.cc:2362] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5. Re-sending same reply.
I20260806 03:52:11.931507 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.925679 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.932698 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.925838 2484 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:11.921814 2331 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:11.933383 2474 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:11.938050 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.940738 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.940944 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.941510 2481 raft_consensus.cc:2362] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7. Re-sending same reply.
I20260806 03:52:11.803225 5834 raft_consensus.cc:3032] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:11.816758 6136 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:11.828758 6180 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:11.948817 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.950138 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.948817 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.952806 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.953708 2477 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:11.883939 6086 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:11.957554 2370 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:11.861434 5886 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:11.962116 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:11.966917 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.918900 3586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 413) took 1022 ms. Trace:
I20260806 03:52:11.972962 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.973222 3586 rpcz_store.cc:276] 0806 03:52:10.896282 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:11.748174 (+851892us) service_pool.cc:227] Handling call
0806 03:52:11.918818 (+170644us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:11.974490 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:11.973708 2481 raft_consensus.cc:2362] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8. Re-sending same reply.
I20260806 03:52:11.976266 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:11.976450 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.977721 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.978010 3590 raft_consensus.cc:1226] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 7. Ops: []
I20260806 03:52:11.978850 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:11.979393 2481 raft_consensus.cc:2362] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5. Re-sending same reply.
I20260806 03:52:11.929258 5933 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.980643 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.981586 2481 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
W20260806 03:52:11.983237 6090 consensus_meta.cc:287] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.560s user 0.006s sys 0.009s
I20260806 03:52:11.983071 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.983559 5933 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:11.984148 2481 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:11.985580 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.934473 6084 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:11.987454 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:11.943867 6205 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:11.817200 5965 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:11.988399 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:11.975348 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 307) took 1185 ms. Trace:
I20260806 03:52:11.970197 2429 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:11.990281 2376 rpcz_store.cc:276] 0806 03:52:10.790347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:10.790532 (+ 185us) service_pool.cc:227] Handling call
0806 03:52:11.974322 (+1183790us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:52:11.975255 (+ 933us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4763904}
I20260806 03:52:11.975338 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:11.991458 2332 leader_election.cc:334] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.992s (SENT)
I20260806 03:52:12.005992 2332 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:11.989938 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:11.943223 5917 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:11.997531 5965 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:11.998312 3591 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:12.000247 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:12.010814 2481 raft_consensus.cc:2362] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5. Re-sending same reply.
W20260806 03:52:12.001919 6192 consensus_meta.cc:287] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.999s user 0.015s sys 0.005s
I20260806 03:52:12.002521 6183 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:11.929105 5888 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:11.995513 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:12.022497 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:11.948053 5834 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:11.929352 5932 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.036985 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:12.010097 6202 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.581s user 0.006s sys 0.016s
W20260806 03:52:12.027592 6068 consensus_meta.cc:287] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.625s user 0.009s sys 0.005s
I20260806 03:52:12.035874 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:11.882791 6227 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.060550 6227 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:12.062744 5833 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:12.050753 2584 raft_consensus.cc:2354] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 6.
I20260806 03:52:12.067994 5888 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:12.068678 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1040) took 1166 ms. Trace:
I20260806 03:52:12.069149 2578 rpcz_store.cc:276] 0806 03:52:10.902490 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:10.902633 (+ 143us) service_pool.cc:227] Handling call
0806 03:52:12.068375 (+1165742us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b540024d6d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:12.068569 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1100486912}
I20260806 03:52:11.983819 6090 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.072630 6090 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.072906 6201 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:12.074994 2478 raft_consensus.cc:1226] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:12.078380 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.079269 2478 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:12.057052 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:12.046151 6183 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:12.054803 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:12.085765 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.086896 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:12.056468 6121 consensus_meta.cc:287] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.093s user 0.018s sys 0.001s
I20260806 03:52:12.057050 3588 raft_consensus.cc:2354] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 9.
I20260806 03:52:12.046152 5964 consensus_queue.cc:1046] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260806 03:52:12.057775 6087 raft_consensus.cc:3032] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:12.084304 6183 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:12.086512 5834 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.023687 6051 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:12.034003 5917 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.004441 6232 consensus_queue.cc:1046] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 3, Time since last communication: 0.138s
I20260806 03:52:12.123040 5834 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:12.123772 6234 log.cc:924] Time spent T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
W20260806 03:52:12.124097 6161 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.601s user 0.010s sys 0.017s
I20260806 03:52:12.124549 6161 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.127350 6161 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.128978 6232 consensus_queue.cc:1225] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer dff94100ffc5485f92cbe6a85addf0b2 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 03:52:12.129788 6232 raft_consensus.cc:3032] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:12.130290 6232 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:12.131597 6194 raft_consensus.cc:3032] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:12.037068 6192 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.098614 6087 raft_consensus.cc:729] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:12.106918 6183 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:12.135124 6185 consensus_meta.cc:287] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.867s user 0.024s sys 0.000s
I20260806 03:52:12.088254 6202 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:12.135685 6194 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:12.054100 6068 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.095160 6121 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.181655 6121 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.131438 6232 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:12.182832 6163 consensus_meta.cc:287] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.745s user 0.008s sys 0.008s
I20260806 03:52:12.182974 6232 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:12.168793 6194 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:12.172549 6233 log.cc:924] Time spent T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260806 03:52:12.178072 6202 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.149550 6185 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:12.120219 6167 consensus_meta.cc:287] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.702s user 0.013s sys 0.006s
I20260806 03:52:12.199222 6192 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:12.192185 6167 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.208209 6167 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:52:12.205483 6185 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.222879 6183 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.271227 2377 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260806 03:52:12.276815 2377 pending_rounds.cc:76] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 4
W20260806 03:52:12.273221 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 302760us
I20260806 03:52:12.277535 2377 pending_rounds.cc:97] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.5
I20260806 03:52:12.289565 2377 raft_consensus.cc:2894] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Aborting config change with OpId 6.5: Aborted: Op aborted by new leader
I20260806 03:52:12.285115 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:12.297061 2474 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:12.298012 2377 raft_consensus.cc:2866] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:12.297160 2370 raft_consensus.cc:2354] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8.
I20260806 03:52:12.298803 2476 raft_consensus.cc:2354] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6.
I20260806 03:52:12.299925 2480 raft_consensus.cc:1226] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
W20260806 03:52:12.300220 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 328798us
I20260806 03:52:12.300122 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.301627 2482 raft_consensus.cc:1226] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:12.303982 2537 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:11.158006 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:52:12.304886 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.306967 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:12.282665 2332 leader_election.cc:334] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.029s (SENT)
I20260806 03:52:12.310302 2572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2219) took 2503 ms. Trace:
I20260806 03:52:12.100044 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:12.313557 3594 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
W20260806 03:52:12.317039 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 296619us
I20260806 03:52:12.269212 2235 catalog_manager.cc:5495] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:12.173168 5917 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:12.176098 5932 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.364832 6143 raft_consensus.cc:3032] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:12.365418 6143 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:12.365447 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 61988.2us
I20260806 03:52:12.368402 2478 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:12.369463 6205 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:12.370136 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.370565 6205 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.372797 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2128) took 1137 ms. Trace:
I20260806 03:52:12.373404 2479 rpcz_store.cc:276] 0806 03:52:11.235489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:11.235906 (+ 417us) service_pool.cc:227] Handling call
0806 03:52:11.409759 (+173853us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:11.417010 (+ 7251us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.372151 (+955141us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b5400037350. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:12.372503 (+ 352us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269559680}
I20260806 03:52:12.375380 6086 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:12.377326 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 95144us
I20260806 03:52:12.160477 6087 consensus_queue.cc:260] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:12.378595 2332 leader_election.cc:334] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.177s (SENT)
I20260806 03:52:12.379232 2332 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:12.379773 6087 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:12.183386 6163 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.382784 2577 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:12.383437 2575 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:12.386456 3591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 411) took 1509 ms. Trace:
I20260806 03:52:12.387053 3591 rpcz_store.cc:276] 0806 03:52:10.876567 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:10.876747 (+ 180us) service_pool.cc:227] Handling call
0806 03:52:12.385870 (+1509123us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54004a2254. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:12.386140 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2477241216}
I20260806 03:52:12.390244 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.333428 2572 rpcz_store.cc:276] 0806 03:52:09.806846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:09.807434 (+ 588us) service_pool.cc:227] Handling call
0806 03:52:09.834720 (+ 27286us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:10.359054 (+524334us) raft_consensus.cc:1520] Early marking committed up to index 1
0806 03:52:10.359100 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:10.366322 (+ 7222us) log.cc:842] Serialized 52 byte log entry
0806 03:52:10.659757 (+293435us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:52:10.659795 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:10.659878 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:12.199905 (+1540027us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b2400227eb8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d799f5cf4 00007f0d799f3dfb 00007f0d79a82559 00007f0d79a821e5 00007f0d79a81740 00007f0d7c774578 00007f0d7c774cd8 00007f0d7c74c6e8 00007f0d7c747aea 00007f0d7c746f36
0806 03:52:12.309252 (+109347us) raft_consensus.cc:1671] finished
0806 03:52:12.309316 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:12.309987 (+ 671us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":626055,"raft.queue_time_us":127418,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":567121,"spinlock_wait_cycles":6523136,"thread_start_us":699991,"threads_started":2,"wal-append.queue_time_us":558621}
I20260806 03:52:12.376648 6086 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.342553 6090 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:12.387595 6183 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:12.371863 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.404618 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.181648 6068 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.414855 2537 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:12.416204 6202 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:12.417397 6202 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.422708 6136 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.423197 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.423503 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.352286 6161 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:12.428401 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:12.428740 6161 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.429530 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:12.430963 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:12.431138 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.431475 6161 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.424755 6090 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.363772 6185 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:12.433488 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:12.437512 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 92706.2us
I20260806 03:52:12.438433 2479 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:12.439208 2534 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:12.439487 6194 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:12.442116 5922 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:12.442384 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 135884us
I20260806 03:52:12.443387 6151 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:12.409456 2481 raft_consensus.cc:2362] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8. Re-sending same reply.
I20260806 03:52:12.444974 2478 raft_consensus.cc:2354] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6.
I20260806 03:52:12.446460 2474 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:12.367405 6143 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.445858 2537 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:12.456147 6051 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:12.456981 6163 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:12.457728 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.458943 2480 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:52:12.459340 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 406169us
I20260806 03:52:12.483501 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 1968) took 1752 ms. Trace:
I20260806 03:52:12.459939 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:12.481212 3594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 410) took 1611 ms. Trace:
I20260806 03:52:12.469884 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.491778 2482 rpcz_store.cc:276] 0806 03:52:10.731107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:10.731245 (+ 138us) service_pool.cc:227] Handling call
0806 03:52:10.731592 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:12.338578 (+1606986us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.483078 (+144500us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b5400388bd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:12.483407 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3609083264}
I20260806 03:52:12.463994 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.502357 2480 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:12.496085 3594 rpcz_store.cc:276] 0806 03:52:10.870069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:10.870269 (+ 200us) service_pool.cc:227] Handling call
0806 03:52:12.480806 (+1610537us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54005424d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:12.481117 (+ 311us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3250680064}
I20260806 03:52:12.470319 2235 catalog_manager.cc:5495] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 2 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 3 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:12.506215 5829 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:12.509485 6163 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:12.510154 6163 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:12.512933 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 73993.8us
I20260806 03:52:12.514564 6203 raft_consensus.cc:2781] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:12.515201 6203 raft_consensus.cc:686] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:12.516558 6141 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:52:12.510402 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.474107 6143 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:12.529346 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.478991 6084 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.463604 6090 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.531489 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.535383 2481 raft_consensus.cc:2454] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:12.476204 6167 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:12.542424 2478 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:12.542896 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:12.543737 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.544749 2478 raft_consensus.cc:2379] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:12.545817 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.547279 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:12.549643 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 80930.5us
I20260806 03:52:12.554626 2479 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:12.558291 2474 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:12.553697 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.549636 6163 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:12.542900 2477 consensus_meta.cc:287] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.589s user 0.017s sys 0.005s
I20260806 03:52:12.563674 6016 consensus_queue.cc:1046] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260806 03:52:12.564833 6086 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.564101 2477 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:12.568142 2484 raft_consensus.cc:2379] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:12.569149 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.572302 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:12.575008 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:12.578668 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 200209us
I20260806 03:52:12.581651 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:12.582031 6086 consensus_queue.cc:1046] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260806 03:52:12.582671 6013 consensus_meta.cc:287] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.982s user 0.007s sys 0.014s
I20260806 03:52:12.583611 6086 raft_consensus.cc:3032] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:12.580282 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:12.584697 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 139146us
I20260806 03:52:12.578181 5755 raft_consensus.cc:3032] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:52:12.584136 6249 log.cc:924] Time spent T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.093s user 0.000s sys 0.002s
I20260806 03:52:12.587204 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2148) took 1004 ms. Trace:
I20260806 03:52:12.586182 2483 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:12.584301 6202 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:12.588951 2477 rpcz_store.cc:276] 0806 03:52:11.583344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:11.951005 (+367661us) service_pool.cc:227] Handling call
0806 03:52:12.580193 (+629188us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:12.587850 2474 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:12.592610 2372 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:12.536582 6185 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.590703 2482 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:12.590025 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2154) took 1003 ms. Trace:
I20260806 03:52:12.596514 2483 rpcz_store.cc:276] 0806 03:52:11.586315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.479739 (+893424us) service_pool.cc:227] Handling call
0806 03:52:12.589972 (+110233us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:12.599321 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.569250 2235 catalog_manager.cc:5495] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 3 to 5, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 5 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:12.583487 6013 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.588335 6205 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:12.589339 5755 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:12.587090 6086 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:12.611078 2479 raft_consensus.cc:1261] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:52:12.612422 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2157) took 1024 ms. Trace:
I20260806 03:52:12.619715 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:12.621440 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:12.622404 2484 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:12.624905 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2143) took 1039 ms. Trace:
I20260806 03:52:12.615782 2481 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:12.616711 2478 rpcz_store.cc:276] 0806 03:52:11.587598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.563180 (+975582us) service_pool.cc:227] Handling call
0806 03:52:12.612329 (+ 49149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:12.629534 2479 rpcz_store.cc:276] 0806 03:52:11.575646 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.567034 (+991388us) service_pool.cc:227] Handling call
0806 03:52:12.595799 (+ 28765us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:12.614227 (+ 18428us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.614517 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:12.612084 6136 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.639045 2537 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:12.595711 6203 consensus_queue.cc:237] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:12.660974 6180 consensus_meta.cc:287] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.670s user 0.011s sys 0.010s
I20260806 03:52:12.661839 6180 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.663511 2476 raft_consensus.cc:2379] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:12.665382 2374 raft_consensus.cc:1261] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 03:52:12.667402 2575 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 03:52:12.667024 6086 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.668516 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 254) took 1071 ms. Trace:
I20260806 03:52:12.669272 6086 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:12.669014 2575 rpcz_store.cc:276] 0806 03:52:11.597058 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:11.597200 (+ 142us) service_pool.cc:227] Handling call
0806 03:52:11.597619 (+ 419us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:12.382118 (+784499us) spinlock_profiling.cc:243] Waited 684 ms on lock 0x7b58006a67b0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7ca40f63 00007f0d7caa4aba 00007f0d7cabc3af 00007f0d7cabdb4e 00007f0d7cac7c89 00007f0d7cac8556 00007f0d7cac4edf 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684
0806 03:52:12.667938 (+285820us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.668417 (+ 479us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1573443968}
I20260806 03:52:12.673112 6185 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.672693 5755 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:12.674072 6164 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:12.674583 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 69578.4us
I20260806 03:52:12.675019 5755 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:12.677591 3542 leader_election.cc:398] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8.
I20260806 03:52:12.678081 3542 leader_election.cc:401] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260806 03:52:12.678815 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 46914.1us
I20260806 03:52:12.682799 6121 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:12.684430 5872 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260806 03:52:12.643698 6087 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:12.691619 2572 raft_consensus.cc:1226] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 3. Current term is 4. Ops: []
W20260806 03:52:12.671841 2332 leader_election.cc:334] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.612s (SENT)
I20260806 03:52:12.701215 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:12.701923 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:12.703001 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:12.703195 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:12.709252 2583 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:12.710769 2332 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:12.707665 2478 raft_consensus.cc:1226] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:12.711382 2477 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:52:12.725085 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:12.726943 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 140132us
I20260806 03:52:12.728395 6205 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:12.728610 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:12.730518 2482 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:52:12.731541 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2155) took 1144 ms. Trace:
I20260806 03:52:12.733143 2482 rpcz_store.cc:276] 0806 03:52:11.586773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.722539 (+1135766us) service_pool.cc:227] Handling call
0806 03:52:12.729959 (+ 7420us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:12.731131 (+ 1172us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.731417 (+ 286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:12.732282 2584 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:12.736755 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:12.735601 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:12.755278 2480 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:12.757054 2483 raft_consensus.cc:1261] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 03:52:12.761251 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2161) took 1134 ms. Trace:
I20260806 03:52:12.765422 2235 catalog_manager.cc:5495] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 5 to 6, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 6 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } }
W20260806 03:52:12.770857 6248 log.cc:924] Time spent T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
W20260806 03:52:12.773016 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 101217us
I20260806 03:52:12.773782 6205 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:12.780527 6087 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.792205 2483 pending_rounds.cc:76] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 2
I20260806 03:52:12.793465 2477 rpcz_store.cc:276] 0806 03:52:11.589373 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.635091 (+1045718us) service_pool.cc:227] Handling call
0806 03:52:12.705447 (+ 70356us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:12.723024 (+ 17577us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.723318 (+ 294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:12.802304 2483 pending_rounds.cc:97] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:52:12.802846 2483 raft_consensus.cc:2866] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:12.803822 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2159) took 1215 ms. Trace:
I20260806 03:52:12.804279 2483 rpcz_store.cc:276] 0806 03:52:11.588613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.756128 (+1167515us) service_pool.cc:227] Handling call
0806 03:52:12.756618 (+ 490us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:12.803466 (+ 46848us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.803751 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:12.805420 2476 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:12.807482 2479 raft_consensus.cc:1261] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260806 03:52:12.807971 2479 pending_rounds.cc:76] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 1
I20260806 03:52:12.808269 2479 pending_rounds.cc:97] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 03:52:12.808528 2479 raft_consensus.cc:2866] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:12.810789 2482 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:12.800791 6180 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:12.814301 2479 pending_rounds.cc:97] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:52:12.815083 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2175) took 1063 ms. Trace:
I20260806 03:52:12.813356 2481 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:12.818652 2479 raft_consensus.cc:2866] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:12.820279 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:12.820255 2476 rpcz_store.cc:276] 0806 03:52:11.751391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.774598 (+1023207us) service_pool.cc:227] Handling call
0806 03:52:12.798229 (+ 23631us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:12.807487 (+ 9258us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.814624 (+ 7137us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b5400350cd0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:12.814835 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":40559360}
I20260806 03:52:12.834288 2478 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:12.835955 2235 catalog_manager.cc:5495] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 5, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 5 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:12.854188 2483 raft_consensus.cc:1226] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:12.856541 2235 catalog_manager.cc:5495] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 2 to 4, leader changed from cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 4 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:12.839090 6016 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260806 03:52:12.860194 6013 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:12.881798 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:12.892120 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:12.893334 2480 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:12.901314 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:12.902971 6044 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:12.903798 6044 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:12.904623 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:12.905071 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 187415us
I20260806 03:52:12.906828 6044 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:12.907096 2236 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33778 (request call id 216) took 11947 ms (client timeout 15000 ms). Trace:
I20260806 03:52:12.907718 2480 raft_consensus.cc:1226] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:12.909942 3594 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:12.910993 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:12.925335 2483 raft_consensus.cc:1261] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260806 03:52:12.907686 2236 rpcz_store.cc:269] 0806 03:52:00.959798 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:00.974094 (+ 14296us) service_pool.cc:227] Handling call
0806 03:52:12.906579 (+11932485us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:52:04.808299 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:52:04.896607 (+ 88308us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:52:04.896670 (+ 63us) write_op.cc:435] Acquired schema lock
0806 03:52:04.896704 (+ 34us) tablet.cc:660] Decoding operations
0806 03:52:04.971132 (+ 74428us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:52:04.971236 (+ 104us) write_op.cc:641] Partition lock acquired for write op
0806 03:52:04.971267 (+ 31us) tablet.cc:683] Acquiring locks for 21 operations
0806 03:52:04.972968 (+ 1701us) tablet.cc:699] Row locks acquired
0806 03:52:04.973011 (+ 43us) write_op.cc:260] PREPARE: finished
0806 03:52:04.973187 (+ 176us) write_op.cc:270] Start()
0806 03:52:04.973359 (+ 172us) write_op.cc:276] Timestamp: P: 1785988324973126 usec, L: 0
0806 03:52:04.973400 (+ 41us) op_driver.cc:331] REPLICATION: starting
0806 03:52:05.059796 (+ 86396us) log.cc:842] Serialized 8315 byte log entry
0806 03:52:05.252163 (+192367us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:05.680262 (+428099us) op_driver.cc:444] REPLICATION: finished
0806 03:52:06.215485 (+535223us) write_op.cc:301] APPLY: starting
0806 03:52:06.235429 (+ 19944us) tablet.cc:1365] starting BulkCheckPresence
0806 03:52:06.284689 (+ 49260us) tablet.cc:1368] finished BulkCheckPresence
0806 03:52:06.284718 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:52:06.689837 (+405119us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:52:06.855019 (+165182us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=21
0806 03:52:06.855086 (+ 67us) write_op.cc:312] APPLY: finished
0806 03:52:06.855657 (+ 571us) log.cc:842] Serialized 145 byte log entry
0806 03:52:07.130348 (+274691us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:52:07.138264 (+ 7916us) write_op.cc:454] Released schema lock
0806 03:52:07.138424 (+ 160us) write_op.cc:341] FINISH: Updating metrics
0806 03:52:07.150455 (+ 12031us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:52:07.150489 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26141,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":440886,"apply.run_cpu_time_us":43446,"apply.run_wall_time_us":963655,"num_ops":21,"prepare.queue_time_us":620727,"prepare.run_cpu_time_us":21764,"prepare.run_wall_time_us":434131,"raft.queue_time_us":181374,"raft.run_cpu_time_us":17955,"raft.run_wall_time_us":464898,"replication_time_us":693414,"spinlock_wait_cycles":10651904,"thread_start_us":864951,"threads_started":5,"wal-append.queue_time_us":437077,"wal-append.run_cpu_time_us":25168,"wal-append.run_wall_time_us":2456986}]]}
I20260806 03:52:12.938405 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:12.944276 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:12.977044 2578 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:12.944923 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:12.945631 3591 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:12.946416 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:12.979313 3588 raft_consensus.cc:2354] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7.
I20260806 03:52:12.949069 5682 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:12.980464 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:12.981268 2572 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:12.935592 2478 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260806 03:52:12.956343 6087 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:12.958222 3594 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260806 03:52:12.960158 2370 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:12.961172 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2172) took 1210 ms. Trace:
I20260806 03:52:12.985019 2479 rpcz_store.cc:276] 0806 03:52:11.750487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.641141 (+890654us) service_pool.cc:227] Handling call
0806 03:52:12.711628 (+ 70487us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:12.960898 (+249270us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:12.961089 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:12.964542 2481 raft_consensus.cc:1226] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:12.966166 2480 raft_consensus.cc:1226] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:12.966353 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:12.967758 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:12.950973 3545 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:12.984402 3594 pending_rounds.cc:76] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Aborting all ops after (but not including) 4
I20260806 03:52:12.998250 3594 pending_rounds.cc:97] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 6.5
I20260806 03:52:12.998624 3594 raft_consensus.cc:2894] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Aborting config change with OpId 6.5: Aborted: Op aborted by new leader
I20260806 03:52:13.001355 2479 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:12.999383 2584 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:13.011727 2482 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:13.014369 2375 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:13.029932 2578 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:13.027907 3594 raft_consensus.cc:2866] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:13.036798 6192 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:13.045639 2479 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:52:13.053586 2538 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:13.054960 6151 consensus_queue.cc:1046] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:13.056393 5922 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:13.056841 5922 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:13.057232 5922 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:13.061707 2534 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:13.070381 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:13.041883 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:13.072997 3588 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:13.055109 6013 consensus_queue.cc:1046] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:52:13.021405 6087 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:13.094813 2484 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:13.096516 2474 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:13.097584 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:13.101948 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48468.1us
W20260806 03:52:13.102660 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 151904us
I20260806 03:52:13.103763 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:13.115329 2372 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.511s user 0.011s sys 0.000s
I20260806 03:52:13.115823 2372 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:13.117127 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2144) took 1541 ms. Trace:
I20260806 03:52:13.120177 2538 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:13.117534 2474 rpcz_store.cc:276] 0806 03:52:11.576003 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.599362 (+1023359us) service_pool.cc:227] Handling call
0806 03:52:13.096255 (+496893us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:13.104881 (+ 8626us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:13.116922 (+ 12041us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b5400350050. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:13.117063 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1073109888}
I20260806 03:52:13.124987 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:13.134050 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:13.136734 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:13.135996 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:13.144151 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:13.151767 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:13.126824 6121 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8.
I20260806 03:52:13.154368 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:13.154623 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:13.155664 6167 consensus_queue.cc:1046] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:13.151422 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:13.158265 5510 consensus_queue.cc:1046] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:52:13.161252 6021 consensus_meta.cc:287] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.310s user 0.016s sys 0.007s
I20260806 03:52:13.163300 2574 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:13.136947 6016 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:13.166108 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:13.167441 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:13.136622 6180 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:13.154410 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:13.187476 2575 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:13.194450 2332 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:13.192936 3547 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:13.208693 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:13.179833 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:13.139745 6087 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:13.222083 2581 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:13.259930 5474 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:13.260782 5474 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:13.262012 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 50588us
I20260806 03:52:13.264925 2578 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
W20260806 03:52:13.271981 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 102218us
W20260806 03:52:13.275381 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 82598.2us
I20260806 03:52:13.237424 5682 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEARNER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:13.267493 3588 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 9: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:13.263442 6051 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:13.219528 6021 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.273290 6167 raft_consensus.cc:3032] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:13.294612 6167 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:13.238353 6180 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.289960 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:13.237823 6194 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.040s
W20260806 03:52:13.300582 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 111931us
I20260806 03:52:13.302597 6180 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:13.302717 2332 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:13.308269 2436 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:13.313058 5474 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:13.314121 6180 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:13.318117 6087 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:13.319387 6121 consensus_queue.cc:1046] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:13.322548 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:13.324476 6084 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:13.325176 6084 raft_consensus.cc:686] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:13.325973 6021 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:13.326247 5828 raft_consensus.cc:3032] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:13.326700 5828 raft_consensus.cc:729] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:13.326818 2479 raft_consensus.cc:1226] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:13.332649 6087 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:13.339162 5884 raft_consensus.cc:3032] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:13.339473 6087 consensus_queue.cc:1046] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:13.341683 5884 raft_consensus.cc:729] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:13.340794 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:13.291360 6051 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:13.351544 5474 raft_consensus.cc:3032] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:13.352185 5474 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:13.352550 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:13.353721 3584 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:13.342834 5884 consensus_queue.cc:260] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.311571 5922 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.350044 6051 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:13.365118 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:13.384775 2581 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:13.353613 5474 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:13.357957 6194 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:13.295677 6167 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.393304 2572 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:13.389241 3590 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:13.326719 6084 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:13.327766 5828 consensus_queue.cc:260] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.400401 6163 raft_consensus.cc:2741] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
W20260806 03:52:13.409392 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 53052.9us
I20260806 03:52:13.407078 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:13.424124 2375 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:13.401506 6194 raft_consensus.cc:686] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:13.434504 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:13.426020 5884 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:13.399724 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 119896us
I20260806 03:52:13.397336 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:13.399648 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 242214us
I20260806 03:52:13.423506 5828 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:13.429299 6203 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:13.402202 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 230855us
I20260806 03:52:13.471271 2538 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:13.438832 5474 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:13.456720 2370 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:13.473763 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:13.474695 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 214645us
W20260806 03:52:13.496241 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 155568us
I20260806 03:52:13.504397 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:13.510807 2377 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:52:13.511919 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:13.506646 2375 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:13.474253 6016 raft_consensus.cc:686] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:13.520941 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 49847.5us
W20260806 03:52:13.523972 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 272917us
I20260806 03:52:13.519248 2572 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
W20260806 03:52:13.527195 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 98471.9us
I20260806 03:52:13.529299 2537 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:13.530345 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:13.531243 2372 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:13.534750 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:13.542557 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:13.522615 6087 raft_consensus.cc:3032] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:13.546343 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:13.484468 6203 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:13.532506 5872 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:13.566999 2537 leader_election.cc:398] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7.
W20260806 03:52:13.570459 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 53414.5us
I20260806 03:52:13.574159 6016 consensus_queue.cc:237] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:13.576975 6276 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260806 03:52:13.578702 2436 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:13.552618 5922 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:13.550239 6087 raft_consensus.cc:729] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:13.568236 5510 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:13.555260 5872 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:13.585297 2370 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:13.582573 6192 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:13.600590 6180 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:13.600993 6192 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:13.601400 6180 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.597258 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:13.604827 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:13.612425 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:13.613556 2575 raft_consensus.cc:2362] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7. Re-sending same reply.
I20260806 03:52:13.603349 5872 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:13.621208 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:13.624106 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:13.628547 6294 log.cc:924] Time spent T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.058s user 0.000s sys 0.001s
I20260806 03:52:13.640856 6167 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:13.641870 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 289) took 1246 ms. Trace:
I20260806 03:52:13.642196 6299 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:13.642807 6299 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:13.643321 6299 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:13.642467 2476 rpcz_store.cc:276] 0806 03:52:12.400077 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:12.876639 (+476562us) service_pool.cc:227] Handling call
0806 03:52:12.890449 (+ 13810us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:12.968503 (+ 78054us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:12.968537 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:13.042574 (+ 74037us) log.cc:842] Serialized 52 byte log entry
0806 03:52:13.445511 (+402937us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:13.539279 (+ 93768us) log.cc:842] Serialized 14 byte log entry
0806 03:52:13.540430 (+ 1151us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:13.540565 (+ 135us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:13.641590 (+101025us) raft_consensus.cc:1671] finished
0806 03:52:13.641614 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:13.641782 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":52944,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":520,"raft.run_cpu_time_us":419,"raft.run_wall_time_us":34627,"thread_start_us":343777,"threads_started":2,"wal-append.queue_time_us":460301}
I20260806 03:52:13.644749 6028 raft_consensus.cc:3032] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:13.645149 6028 raft_consensus.cc:729] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:13.624349 2579 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
W20260806 03:52:13.681087 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 61397.7us
I20260806 03:52:13.560237 6194 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:13.678860 6021 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:13.692557 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:13.596715 6293 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.098s
I20260806 03:52:13.617769 5755 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.073s
W20260806 03:52:13.704543 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 139040us
W20260806 03:52:13.710122 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 141193us
W20260806 03:52:13.719210 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 165421us
I20260806 03:52:13.720562 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:13.722640 2579 raft_consensus.cc:1226] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:13.724997 6298 consensus_queue.cc:1046] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.081s
I20260806 03:52:13.726389 6298 raft_consensus.cc:3032] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:13.726781 6298 raft_consensus.cc:729] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:13.707152 6028 consensus_queue.cc:260] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.732980 6203 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:13.733428 6028 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:13.712319 6180 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:13.736276 6297 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:13.736791 6297 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:13.737221 6297 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:13.745213 5884 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:13.745999 5884 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:13.747071 6302 log.cc:924] Time spent T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.111s user 0.000s sys 0.002s
I20260806 03:52:13.708062 6087 consensus_queue.cc:260] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.750330 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:13.752449 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:13.754016 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:13.753912 6303 consensus_queue.cc:1046] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:52:13.764609 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:13.763008 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 206378us
I20260806 03:52:13.768813 6303 raft_consensus.cc:3032] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:13.769325 6303 raft_consensus.cc:729] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:13.770422 6303 consensus_queue.cc:260] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.771271 6087 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:13.764460 2429 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:13.778308 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 63540.6us
W20260806 03:52:13.779691 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 55518.9us
I20260806 03:52:13.780664 3547 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:13.785750 6180 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:13.786450 6180 raft_consensus.cc:686] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:13.787961 2436 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:13.787468 6180 consensus_queue.cc:237] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:13.790849 2436 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:13.795222 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:52:13.757974 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0862715928815305s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2203s for a response
I20260806 03:52:13.794656 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:13.771891 6303 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:13.850997 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 70449.8us
I20260806 03:52:13.853984 3590 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
W20260806 03:52:13.855847 2332 leader_election.cc:334] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.526s (SENT)
I20260806 03:52:13.796885 6021 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:13.868700 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 68047us
W20260806 03:52:13.869346 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 70666.9us
W20260806 03:52:13.871727 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 81028.9us
I20260806 03:52:13.872648 2377 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:13.856941 2332 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:13.874650 6287 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:13.875355 6287 raft_consensus.cc:686] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:13.836673 5682 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:13.877553 5682 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:13.877985 5682 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:13.882673 6316 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:13.800181 5829 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:13.802333 5884 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:13.901360 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 62407.7us
I20260806 03:52:13.901793 3545 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:13.727861 6298 consensus_queue.cc:260] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:13.921818 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 127679us
I20260806 03:52:13.924544 6126 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.119s
I20260806 03:52:13.930735 5829 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:13.822410 5872 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:13.998867 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 83087.3us
I20260806 03:52:13.932297 6084 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:13.918407 6205 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:14.002132 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 164112us
I20260806 03:52:13.872278 6051 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:14.004884 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 152233us
I20260806 03:52:14.009672 2370 raft_consensus.cc:1261] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 03:52:14.006928 6028 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:14.024703 2332 leader_election.cc:334] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.588s (SENT)
I20260806 03:52:14.028930 6194 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:14.041498 6051 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:14.042024 6194 raft_consensus.cc:686] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:14.043817 6203 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:14.044593 6203 consensus_queue.cc:1225] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer 608e57d72aae496aaf8613622153aa0c log is divergent from this leader: its last log entry 5.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 03:52:14.029661 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:14.030666 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:14.038331 5682 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:14.053529 6028 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.054459 5682 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:14.050236 6192 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:14.058280 2377 raft_consensus.cc:1226] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:14.043226 6194 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:14.075730 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58259.9us
I20260806 03:52:14.092213 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:14.055735 6028 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:14.067536 2332 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.122627 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:14.124066 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:14.124866 2538 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:14.010664 6205 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.129498 6028 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:14.131340 5828 consensus_queue.cc:1046] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:14.132768 6203 raft_consensus.cc:3032] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:14.133168 5829 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:14.135057 2476 raft_consensus.cc:3037] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:14.137010 6311 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:14.131701 2436 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.033932 5830 consensus_queue.cc:1046] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.088s
I20260806 03:52:14.141467 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:14.142292 3590 raft_consensus.cc:2362] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8. Re-sending same reply.
I20260806 03:52:14.143940 6016 consensus_queue.cc:1035] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:14.148070 2477 raft_consensus.cc:1226] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:14.153008 6298 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:14.154114 2483 raft_consensus.cc:1261] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:52:14.156424 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:14.157114 2483 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:14.157670 5702 consensus_queue.cc:1046] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:14.158087 5872 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:14.017839 6313 consensus_queue.cc:1046] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20260806 03:52:14.159749 5830 raft_consensus.cc:3032] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:14.160211 5830 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:14.160562 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:14.161365 2375 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:14.161464 5830 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.165539 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:14.166548 6313 raft_consensus.cc:3032] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:14.167034 6313 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:14.038316 6299 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:14.168166 6313 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.170753 6299 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:14.174693 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:14.175283 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:14.176229 2484 raft_consensus.cc:2379] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:14.177209 2376 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:14.148797 6028 raft_consensus.cc:686] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:14.179013 6028 consensus_queue.cc:237] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:14.180521 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 164403us
I20260806 03:52:14.181684 6205 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:14.182484 6205 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:14.182811 2483 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:14.183053 6205 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.184424 2372 raft_consensus.cc:1261] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260806 03:52:14.187939 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:14.189028 3587 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:14.189342 2484 pending_rounds.cc:76] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 3
I20260806 03:52:14.189774 2484 pending_rounds.cc:97] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260806 03:52:14.190294 2484 raft_consensus.cc:2866] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:14.191908 2534 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:14.192185 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:14.024407 6287 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:14.192960 5828 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:14.193388 5828 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:14.193778 5828 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:14.199680 2332 leader_election.cc:334] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.601s (SENT)
I20260806 03:52:14.200255 2332 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:14.200911 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 299265us
I20260806 03:52:14.201850 3545 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.204089 2373 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:14.205585 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:14.206518 2574 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:52:14.208873 2583 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260806 03:52:14.217828 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 196287us
I20260806 03:52:14.227945 2476 raft_consensus.cc:1261] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260806 03:52:14.225903 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 94314.3us
I20260806 03:52:14.133217 6203 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:14.254415 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:14.256172 2436 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.223628 6021 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:14.231460 6303 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:14.233903 6126 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:14.241209 5830 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:14.281751 2534 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:14.283035 6313 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:14.298861 2436 leader_election.cc:398] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 6.
I20260806 03:52:14.299314 2436 leader_election.cc:401] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:52:14.300812 2436 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:14.302682 6087 consensus_meta.cc:287] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.531s user 0.009s sys 0.002s
W20260806 03:52:14.304332 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 104688us
W20260806 03:52:14.304998 2332 leader_election.cc:334] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.689s (SENT)
I20260806 03:52:14.305336 2332 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.306494 6192 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:14.307009 6316 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 6.
I20260806 03:52:14.288379 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:14.307997 6316 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:14.308526 3587 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:14.309476 2436 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.311434 2436 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:14.314388 6180 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:14.314467 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:14.315323 6277 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:14.261340 5884 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:14.328791 2436 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.308660 6316 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:14.250404 6299 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
I20260806 03:52:14.337559 2436 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.344169 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:14.344475 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:14.348904 2481 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:14.290192 6021 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:14.308138 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 85077us
W20260806 03:52:14.336784 6297 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.540s user 0.013s sys 0.000s
I20260806 03:52:14.385484 6277 raft_consensus.cc:686] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:14.375420 2379 raft_consensus.cc:1261] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:14.387279 6277 consensus_queue.cc:237] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.392545 6180 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:52:14.401573 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58703.1us
I20260806 03:52:14.284605 5922 consensus_queue.cc:1035] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
W20260806 03:52:14.412192 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 107210us
I20260806 03:52:14.352313 6299 consensus_queue.cc:1225] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260806 03:52:14.426267 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 70760.7us
I20260806 03:52:14.419672 2537 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:14.342748 6316 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:14.399547 6021 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:14.400004 6087 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:14.455682 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:14.338224 5884 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:14.452893 2332 leader_election.cc:334] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.048s (SENT)
I20260806 03:52:14.472153 2332 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.472309 2574 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:14.403751 6297 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:14.485059 6164 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:14.496181 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:52:14.497963 5884 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:14.446677 6141 consensus_queue.cc:1035] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
I20260806 03:52:14.431751 6308 consensus_queue.cc:1035] T 2ccb469db1664d3e941e7a360d048af2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260806 03:52:14.484670 6021 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:52:14.499209 5828 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:14.531406 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 111885us
I20260806 03:52:14.532444 2537 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:14.532289 2481 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:14.508381 6084 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 03:52:14.541324 2374 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:14.541774 6203 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:14.547029 6350 log.cc:924] Time spent T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260806 03:52:14.541788 6164 raft_consensus.cc:686] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:14.511129 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 173578us
I20260806 03:52:14.519248 6205 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:14.584988 2436 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:14.567443 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 111389us
I20260806 03:52:14.588395 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:14.594408 2538 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:14.595679 6343 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260806 03:52:14.596580 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:14.597306 6295 raft_consensus.cc:3032] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:14.597766 6295 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:14.597944 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:14.598738 6295 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.582249 5510 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:14.601085 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:14.602005 2583 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:14.648916 2538 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:14.652010 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 211577us
I20260806 03:52:14.585819 6297 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:14.658166 2579 pending_rounds.cc:76] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 5
I20260806 03:52:14.658560 2579 pending_rounds.cc:97] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260806 03:52:14.658802 2579 raft_consensus.cc:2866] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:14.554903 6013 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.093s
I20260806 03:52:14.587637 6087 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:14.587273 6311 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:14.656157 6084 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:14.667508 6164 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:14.669503 5828 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:14.672184 6354 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260806 03:52:14.682264 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:14.674258 6180 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:14.656760 6051 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:14.677623 2332 leader_election.cc:334] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.060s (SENT)
I20260806 03:52:14.645901 5510 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:14.683877 2332 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.683854 6051 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:14.684080 5510 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:14.685606 2332 leader_election.cc:334] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.139s (SENT)
I20260806 03:52:14.686055 2332 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.686427 6051 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:14.686978 5510 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:14.687134 6051 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:14.688248 6297 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260806 03:52:14.690846 6344 log.cc:924] Time spent T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.147s user 0.002s sys 0.000s
W20260806 03:52:14.687847 2332 leader_election.cc:334] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.183s (SENT)
I20260806 03:52:14.680585 6087 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.653926 5884 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:14.693220 6087 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:14.678148 6311 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.683223 6180 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:14.700945 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:14.662714 6203 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:14.710405 2332 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:14.704537 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 94109.3us
I20260806 03:52:14.665056 6295 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:14.718569 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 165086us
I20260806 03:52:14.722451 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:14.723527 2577 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:14.725056 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:14.735143 2331 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:14.737226 6205 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:14.737833 6205 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:14.738374 6205 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:52:14.746426 6362 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260806 03:52:14.750350 6051 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:14.750774 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 165940us
I20260806 03:52:14.751730 2436 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:14.754017 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 47719.7us
I20260806 03:52:14.755856 3545 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.756552 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:14.757565 6307 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:14.763031 3545 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.764400 6021 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:14.764487 6353 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:52:14.765707 6287 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:14.769639 6311 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:14.771308 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 59112.1us
I20260806 03:52:14.773772 2481 raft_consensus.cc:1261] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:14.764469 6367 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:14.776713 6367 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:14.779222 6367 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:14.712775 6167 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:14.774936 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:14.780570 6167 raft_consensus.cc:686] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:14.713966 6084 raft_consensus.cc:686] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:14.789168 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:14.773280 6298 consensus_meta.cc:287] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.620s user 0.013s sys 0.001s
I20260806 03:52:14.800288 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:14.800727 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 53349.4us
I20260806 03:52:14.801555 2377 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:14.801792 2482 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 6
W20260806 03:52:14.802776 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 364127us
I20260806 03:52:14.798806 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:14.819984 2477 raft_consensus.cc:1261] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:14.821712 2374 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:14.791115 6313 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.500s user 0.012s sys 0.001s
I20260806 03:52:14.829308 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:14.829563 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:14.831135 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:14.833644 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 145827us
I20260806 03:52:14.835377 2483 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:52:14.844790 2377 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:14.839440 2583 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:14.845073 2481 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:14.875029 2482 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:14.825966 6308 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260806 03:52:14.934695 2377 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260806 03:52:14.946894 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 195221us
I20260806 03:52:14.951805 2538 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:14.980764 2436 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:14.981945 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:15.001585 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 147441us
I20260806 03:52:14.970299 6087 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:15.005874 2534 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:15.008975 2538 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:14.946843 6293 consensus_queue.cc:1035] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.055s
I20260806 03:52:14.986982 6313 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:15.002656 5755 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:15.040208 6338 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:15.040372 5755 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.040974 6338 consensus_queue.cc:1225] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer 608e57d72aae496aaf8613622153aa0c log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 03:52:14.975301 6167 consensus_queue.cc:237] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.042639 5755 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:14.945940 6084 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:15.007153 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:15.002511 6021 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:15.033944 6286 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:15.048192 2436 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:15.050495 2374 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:15.010245 5872 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:15.039777 6316 consensus_meta.cc:287] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.581s user 0.011s sys 0.000s
I20260806 03:52:15.052753 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:15.074008 3590 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 03:52:15.073854 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:15.056278 6286 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:15.093806 2332 leader_election.cc:334] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.016s (SENT)
I20260806 03:52:15.094563 2332 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:15.093429 6286 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:15.086665 6311 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260806 03:52:15.101712 6212 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:15.102403 6212 raft_consensus.cc:686] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:15.097151 2484 pending_rounds.cc:76] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 5
I20260806 03:52:15.104336 2484 pending_rounds.cc:97] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260806 03:52:15.104758 2484 raft_consensus.cc:2866] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:15.010929 6087 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:15.116237 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 93825.6us
I20260806 03:52:15.086735 3588 raft_consensus.cc:2379] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:52:15.120250 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 294538us
I20260806 03:52:15.124234 2374 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:52:15.083005 6051 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:15.095209 6021 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:52:15.140431 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 124577us
I20260806 03:52:15.146732 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:15.151265 6338 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:15.151372 6320 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:52:15.103441 6212 consensus_queue.cc:237] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.149116 2483 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
W20260806 03:52:15.159221 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 65511.8us
I20260806 03:52:15.156535 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:15.161036 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:15.161674 2372 raft_consensus.cc:2362] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6. Re-sending same reply.
I20260806 03:52:15.073168 6316 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:15.170176 2369 raft_consensus.cc:1261] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:15.169124 6297 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:15.172004 6297 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.174216 6297 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:15.177736 6087 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:15.158878 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:15.136355 6311 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:15.192082 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:15.196386 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 50861.3us
I20260806 03:52:15.210695 2482 raft_consensus.cc:1226] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:15.169963 6021 raft_consensus.cc:686] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:15.162840 5682 consensus_queue.cc:1046] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.057s
I20260806 03:52:15.225612 2372 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 03:52:15.202037 3584 raft_consensus.cc:1226] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:15.164520 6180 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:15.193717 6013 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:15.208591 6299 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:15.204106 5872 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:15.260696 2332 leader_election.cc:334] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.047s (SENT)
I20260806 03:52:15.139786 6205 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:15.226290 6203 consensus_meta.cc:287] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.510s user 0.013s sys 0.000s
I20260806 03:52:15.280225 2332 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:15.262473 6311 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:15.306058 2474 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:15.227327 6313 consensus_queue.cc:1035] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260806 03:52:15.278895 6194 raft_consensus.cc:3032] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:15.319298 6297 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:15.321192 6194 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
W20260806 03:52:15.339259 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 78620.4us
I20260806 03:52:15.276463 6013 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:15.343283 2332 leader_election.cc:334] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.184s (SENT)
I20260806 03:52:15.344551 2332 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:15.303150 5872 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:15.350176 6398 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:15.350850 6398 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:15.352607 6372 log.cc:924] Time spent T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.371s user 0.000s sys 0.002s
I20260806 03:52:15.353504 6392 consensus_queue.cc:1046] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.038s
I20260806 03:52:15.361722 6392 raft_consensus.cc:3032] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:15.362321 6392 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:15.363289 6392 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:15.348839 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:15.369736 6205 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:15.329315 6021 consensus_queue.cc:237] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:15.373050 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2026) took 1201 ms. Trace:
I20260806 03:52:15.349220 6087 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:15.376863 6393 consensus_queue.cc:1035] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:52:15.316254 6311 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:15.386902 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 43684.8us
I20260806 03:52:15.388155 2538 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:15.353238 6398 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:15.403890 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 138492us
I20260806 03:52:15.356930 6180 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:15.406309 2332 leader_election.cc:334] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.191s (SENT)
I20260806 03:52:15.398590 2479 rpcz_store.cc:276] 0806 03:52:14.171154 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:14.171321 (+ 167us) service_pool.cc:227] Handling call
0806 03:52:14.171701 (+ 380us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:14.484262 (+312561us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:14.484325 (+ 63us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:14.504508 (+ 20183us) log.cc:842] Serialized 52 byte log entry
0806 03:52:14.695048 (+190540us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:14.731130 (+ 36082us) log.cc:842] Serialized 14 byte log entry
0806 03:52:14.851750 (+120620us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:14.851824 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:15.351502 (+499678us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:15.372505 (+ 21003us) raft_consensus.cc:1671] finished
0806 03:52:15.372652 (+ 147us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:15.372873 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86429,"prepare.queue_time_us":26669,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":56,"raft.queue_time_us":121701,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":11643,"spinlock_wait_cycles":48123264,"thread_start_us":536455,"threads_started":3,"wal-append.queue_time_us":354435}
I20260806 03:52:15.412955 2332 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:15.414525 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.414873 2581 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:15.415285 2583 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
W20260806 03:52:15.416939 2436 connection.cc:232] RPC call timeout handler was delayed by -2.7006506519392133s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.198s for a response
W20260806 03:52:15.417520 2436 connection.cc:232] RPC call timeout handler was delayed by -2.6574506517499685s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2028s for a response
I20260806 03:52:15.424738 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.429867 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.433164 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:15.439738 3586 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:15.441063 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:15.405284 6316 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:15.446813 2372 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 7: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:15.424259 6087 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:15.468637 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:15.410817 6194 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:15.469668 3587 raft_consensus.cc:2379] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:15.470175 3587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 423) took 1033 ms. Trace:
W20260806 03:52:15.455791 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 49511us
W20260806 03:52:15.472271 2332 leader_election.cc:334] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.558s (SENT)
I20260806 03:52:15.473053 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:15.460697 5872 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:15.458993 6369 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:15.470527 3587 rpcz_store.cc:276] 0806 03:52:14.436375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:14.440476 (+ 4101us) service_pool.cc:227] Handling call
0806 03:52:15.468495 (+1028019us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b5800000f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65bce 000000000034aea9 00007f0d81991a30 00007f0d81a577b4 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:52:15.470108 (+ 1613us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67085696}
I20260806 03:52:15.445994 6167 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:15.453583 6013 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:15.470192 6194 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:15.471962 6286 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.547852 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:15.549572 2332 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:15.497049 2576 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:15.538069 2331 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:15.483975 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 57876.8us
I20260806 03:52:15.559336 6286 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:15.559983 6084 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:15.560446 5474 consensus_queue.cc:1046] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260806 03:52:15.548890 6121 raft_consensus.cc:659] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: failed to trigger leader election: Illegal state: only voting members can start elections: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:15.492252 6167 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:15.569139 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:15.578995 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:15.582918 2376 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:52:15.513336 6393 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:15.592456 6393 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:15.595188 6393 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:15.494297 6087 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:15.552800 6292 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:15.597492 6087 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:15.601929 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.600872 2331 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:15.602524 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.603461 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.601694 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.604271 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:15.565608 6084 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:15.610618 2537 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:15.585214 6167 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:15.612092 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.615371 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:15.613320 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:15.614243 2574 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260806 03:52:15.486949 6369 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:15.625609 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 78198.9us
I20260806 03:52:15.626343 2576 raft_consensus.cc:1261] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:15.628955 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:15.636047 6205 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:15.636607 6205 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:15.636974 6205 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:15.604951 6311 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:15.597393 6292 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:15.646718 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:15.613617 6084 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:15.602392 6087 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:15.611905 6408 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:15.623970 6369 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:15.653373 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:15.656937 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:15.662633 2369 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:15.664814 2534 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:15.660530 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2262) took 1055 ms. Trace:
I20260806 03:52:15.649508 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:15.677781 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 74826us
I20260806 03:52:15.648692 3587 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260806 03:52:15.681421 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 233605us
I20260806 03:52:15.661290 6087 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:15.668937 2579 rpcz_store.cc:276] 0806 03:52:14.605645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:14.657647 (+ 52002us) service_pool.cc:227] Handling call
0806 03:52:14.657995 (+ 348us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:14.693533 (+ 35538us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:14.693566 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:14.782960 (+ 89394us) spinlock_profiling.cc:243] Waited 6.94 ms on lock 0x7b2c001765a8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:52:14.783533 (+ 573us) log.cc:842] Serialized 52 byte log entry
0806 03:52:14.888439 (+104906us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:14.982996 (+ 94557us) log.cc:842] Serialized 14 byte log entry
0806 03:52:14.983343 (+ 347us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:14.983460 (+ 117us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:15.459024 (+475564us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b1800358cc0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:15.660267 (+201243us) raft_consensus.cc:1671] finished
0806 03:52:15.660287 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:15.660461 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31093,"prepare.queue_time_us":149063,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":31740,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":16994,"spinlock_wait_cycles":21552384,"thread_start_us":137671,"threads_started":2,"wal-append.queue_time_us":220101}
I20260806 03:52:15.694078 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:15.696347 6423 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:15.696589 6424 log.cc:924] Time spent T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.140s user 0.000s sys 0.001s
I20260806 03:52:15.690596 6021 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:15.689586 6051 consensus_queue.cc:1046] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260806 03:52:15.712419 6432 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260806 03:52:15.717144 6051 consensus_queue.cc:1225] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 6.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 03:52:15.716889 6021 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.717839 6051 raft_consensus.cc:3032] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:15.718215 6051 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:15.719249 6021 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:15.719012 6051 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.679700 6408 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.725054 6408 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:15.729059 6167 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:15.731492 6403 log.cc:924] Time spent T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.322s user 0.003s sys 0.000s
W20260806 03:52:15.732038 6402 log.cc:924] Time spent T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.330s user 0.002s sys 0.000s
I20260806 03:52:15.732357 2581 raft_consensus.cc:2354] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:15.732971 6167 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:15.749847 6292 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:15.750852 6425 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:15.751372 6425 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:15.751803 6425 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:15.754760 6021 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:15.757226 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:15.761886 2379 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:52:15.763604 2578 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:15.766244 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:15.767303 2374 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:15.774049 2376 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:15.777031 2534 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:15.779460 2534 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:15.789615 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 251742us
I20260806 03:52:15.791396 2584 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:15.772261 6051 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:15.755774 6021 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:15.792776 6360 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:52:15.809342 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 198870us
I20260806 03:52:15.810571 2537 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:15.807464 2436 connection.cc:232] RPC call timeout handler was delayed by -2.5872506517916918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
W20260806 03:52:15.811753 2436 connection.cc:232] RPC call timeout handler was delayed by -1.3528860518708825s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16200000000000003s for a response
I20260806 03:52:15.817243 2331 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:15.812278 2436 connection.cc:232] RPC call timeout handler was delayed by -1.3204860519617796s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1656s for a response
W20260806 03:52:15.820456 2436 connection.cc:232] RPC call timeout handler was delayed by -1.3015860519371927s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16770000000000002s for a response
I20260806 03:52:15.820827 6430 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:15.802656 5829 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260806 03:52:15.827523 2436 connection.cc:232] RPC call timeout handler was delayed by -1.1134860520251095s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1886s for a response
I20260806 03:52:15.828642 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:15.827684 6432 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.003s
I20260806 03:52:15.832304 6435 consensus_queue.cc:1035] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20260806 03:52:15.838090 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:15.826977 6299 consensus_meta.cc:287] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.534s user 0.016s sys 0.000s
I20260806 03:52:15.821223 6413 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:15.822348 6360 raft_consensus.cc:686] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:15.847352 2376 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260806 03:52:15.847764 2376 pending_rounds.cc:76] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 7
I20260806 03:52:15.847441 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:15.848060 2376 pending_rounds.cc:97] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260806 03:52:15.848454 2376 raft_consensus.cc:2866] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:15.844110 2537 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:15.849920 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:15.849771 6360 consensus_queue.cc:237] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.850373 6432 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:52:15.845479 6430 raft_consensus.cc:686] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:15.859400 6396 consensus_queue.cc:1046] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:15.861809 2480 raft_consensus.cc:2379] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:15.861133 2577 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:52:15.887661 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:15.846665 6299 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:15.893826 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:15.893721 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:15.912114 6448 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:15.912400 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:15.913219 2474 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:15.893549 5474 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260806 03:52:15.916775 5474 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:52:15.919718 6021 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:15.920809 6425 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.921983 6423 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:15.923602 6425 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:15.845515 6205 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.924561 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:15.943066 6205 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:15.924523 6430 consensus_queue.cc:237] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.927685 5829 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:15.944167 5829 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:15.928594 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:15.946887 2369 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:15.949995 6392 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:15.960336 6013 raft_consensus.cc:1050] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
W20260806 03:52:15.930778 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.006s sys 0.002s
W20260806 03:52:15.963137 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.007s sys 0.003s
I20260806 03:52:15.933962 2576 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
W20260806 03:52:15.963507 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 555221us
I20260806 03:52:15.935076 6299 consensus_queue.cc:1035] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260806 03:52:15.869675 6087 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:15.984798 2379 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
W20260806 03:52:15.989766 2332 leader_election.cc:334] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.616s (SENT)
I20260806 03:52:15.990444 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:15.995409 2534 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:16.001073 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.002017 2479 raft_consensus.cc:2379] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.008692 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.009332 2377 raft_consensus.cc:1226] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:16.013499 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.017676 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.018636 2480 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:16.023705 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 134614us
I20260806 03:52:16.027184 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:16.029947 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.033679 2477 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:16.034338 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.050557 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.047618 2369 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:16.055968 2479 raft_consensus.cc:2379] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.057703 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.057813 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.059059 2482 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:16.025328 6448 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:16.058202 2534 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:16.066332 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:16.068898 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 79205.7us
I20260806 03:52:16.024067 3586 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:16.041321 6384 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.036518 6021 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.087890 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.090124 2538 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:16.100528 2477 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:16.103574 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.106101 2480 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:16.108839 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.120414 2484 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.085269 6448 raft_consensus.cc:686] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:16.130300 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.125802 6084 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.131139 2480 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:16.131179 6448 consensus_queue.cc:237] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.132153 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.132959 6084 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:16.133342 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.134429 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.128377 6457 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:16.151769 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:16.152529 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.154342 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.155270 2474 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:16.160223 3547 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:16.159178 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.129410 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.129554 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.173902 2578 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:16.174324 2477 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.095572 6021 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.130373 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.176255 2481 raft_consensus.cc:2379] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.176422 2331 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:16.135128 2480 raft_consensus.cc:2379] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.123059 2482 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:16.177523 6021 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:16.178238 6353 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.178958 6353 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.177533 6084 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:16.181545 6084 raft_consensus.cc:686] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:16.181638 6353 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:16.182777 6084 consensus_queue.cc:237] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.185741 2484 raft_consensus.cc:2379] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:16.175243 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 73020.7us
I20260806 03:52:16.189093 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:16.191843 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:16.199023 2374 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:16.190011 2577 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:16.202312 2331 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:16.202642 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.206920 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.208745 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:16.215215 2474 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:16.217465 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:16.222389 2377 raft_consensus.cc:2396] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:16.223703 6431 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.223939 6341 raft_consensus.cc:3032] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:16.224622 6341 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:16.224352 6431 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:16.225903 6341 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:16.234565 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.235549 2483 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:16.238520 6429 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:16.239176 6429 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:16.239589 6429 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:16.243587 6455 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:16.244546 6455 raft_consensus.cc:686] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:16.246075 6455 consensus_queue.cc:237] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.096998 6384 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.231560 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.221109 6414 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:16.266718 2482 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260806 03:52:16.272729 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.279482 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:16.280258 2377 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:16.284216 3542 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:16.275740 6405 log.cc:924] Time spent T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.926s user 0.005s sys 0.000s
I20260806 03:52:16.287833 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2057) took 1141 ms. Trace:
I20260806 03:52:16.288245 2476 rpcz_store.cc:276] 0806 03:52:15.146173 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:15.147103 (+ 930us) service_pool.cc:227] Handling call
0806 03:52:15.180197 (+ 33094us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:15.180700 (+ 503us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:15.180741 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:15.181382 (+ 641us) log.cc:842] Serialized 26 byte log entry
0806 03:52:15.245125 (+ 63743us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:15.245210 (+ 85us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:15.245391 (+ 181us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:16.274858 (+1029467us) spinlock_profiling.cc:243] Waited 4.81 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:16.287477 (+ 12619us) raft_consensus.cc:1671] finished
0806 03:52:16.287505 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:16.287764 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1368,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":21983,"spinlock_wait_cycles":11053824,"thread_start_us":63099,"threads_started":1,"wal-append.queue_time_us":132520}
I20260806 03:52:16.292807 5682 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:52:16.295006 6384 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:16.332393 2474 raft_consensus.cc:1226] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:16.335060 6441 consensus_queue.cc:1046] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:16.223253 6353 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.127s
I20260806 03:52:16.314349 6021 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:16.347723 6087 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:16.352054 6431 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:16.431900 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.425326 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.471150 2483 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.349001 5829 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.296298 2476 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14fa621f65da481d96d569096ded0a35"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54212
I20260806 03:52:16.488830 2476 raft_consensus.cc:595] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Received request to transfer leadership
I20260806 03:52:16.544981 2477 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.622532 5829 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:16.414376 6021 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.644462 6021 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:16.628360 6340 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:16.646099 6340 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:16.646623 6340 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:16.650781 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:16.600255 6422 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.647s user 0.007s sys 0.019s
I20260806 03:52:16.662297 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.612103 6361 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.701512 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:16.688956 6422 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s user 0.008s sys 0.019s
I20260806 03:52:16.706339 6205 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.705816 6429 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:16.706882 6283 raft_consensus.cc:3032] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:16.708046 6283 raft_consensus.cc:729] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:16.707053 6205 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:16.709254 6429 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:16.716379 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 626354us
W20260806 03:52:16.723429 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 622957us
W20260806 03:52:16.723767 2332 leader_election.cc:334] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.540s (SENT)
I20260806 03:52:16.724279 2332 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:16.725808 2332 leader_election.cc:334] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.666s (SENT)
I20260806 03:52:16.726413 2332 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:16.727968 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.730876 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:16.731660 2377 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:16.731602 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:16.732462 2369 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:52:16.736315 6084 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:16.739878 6205 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:16.744457 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.745383 2477 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:16.750311 6013 consensus_queue.cc:237] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 4, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:16.755077 6474 log.cc:924] Time spent T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.433s user 0.006s sys 0.000s
I20260806 03:52:16.761591 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:16.764652 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 508963us
I20260806 03:52:16.768251 6414 raft_consensus.cc:686] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:16.771461 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:16.772876 6341 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:16.724592 5922 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:16.776798 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58818.1us
I20260806 03:52:16.779326 2478 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260806 03:52:16.791253 2478 raft_consensus.cc:1261] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:16.721674 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.792660 2484 raft_consensus.cc:2362] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7. Re-sending same reply.
I20260806 03:52:16.795594 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:16.820786 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 95022.4us
W20260806 03:52:16.821800 2332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.707s (SENT) [suppressed 26 similar messages]
I20260806 03:52:16.800138 6084 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260806 03:52:16.842386 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 45159.6us
W20260806 03:52:16.843765 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 59107.6us
I20260806 03:52:16.857571 2369 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
W20260806 03:52:16.861016 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 72879.2us
I20260806 03:52:16.878710 2477 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:16.846434 6386 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.704370 6361 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.890599 2436 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:16.744588 6283 consensus_queue.cc:260] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:16.769483 6414 consensus_queue.cc:237] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:16.910633 2578 raft_consensus.cc:1261] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:16.911778 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:16.917310 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:16.921031 3547 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:16.897310 6311 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.942061 6311 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:16.925913 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:16.943879 2478 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:16.945072 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:16.945730 2478 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:16.927889 2481 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:16.949838 2481 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 03:52:16.927395 6481 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:16.928999 6283 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:16.931484 6470 log.cc:924] Time spent T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.603s user 0.000s sys 0.010s
I20260806 03:52:16.938122 6432 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:16.926684 6353 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:16.935555 6044 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:16.951997 2481 raft_consensus.cc:1261] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:16.967221 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:16.974014 2577 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
W20260806 03:52:16.971835 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 81359.5us
I20260806 03:52:16.960331 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2282) took 1120 ms. Trace:
I20260806 03:52:16.977684 2575 rpcz_store.cc:276] 0806 03:52:15.840352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:15.841468 (+ 1116us) service_pool.cc:227] Handling call
0806 03:52:15.853013 (+ 11545us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:15.889121 (+ 36108us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:15.889164 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:15.894404 (+ 5240us) log.cc:842] Serialized 26 byte log entry
0806 03:52:16.355807 (+461403us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:16.355853 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:16.355935 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:16.960092 (+604157us) raft_consensus.cc:1671] finished
0806 03:52:16.960115 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:16.960262 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":17148,"raft.queue_time_us":28926,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":6280,"thread_start_us":415490,"threads_started":1,"wal-append.queue_time_us":921403}
W20260806 03:52:16.980046 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 70349.2us
I20260806 03:52:16.963169 3586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 431) took 1354 ms. Trace:
I20260806 03:52:16.986578 5922 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:16.988739 3586 rpcz_store.cc:276] 0806 03:52:15.608370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:15.613383 (+ 5013us) service_pool.cc:227] Handling call
0806 03:52:16.962975 (+1349592us) spinlock_profiling.cc:243] Waited 332 ms on lock 0x7b54005a05d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:16.963102 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":763351936}
I20260806 03:52:16.989459 2584 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:16.991008 6205 consensus_queue.cc:1035] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:16.954190 6481 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.001600 6481 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:16.902884 6386 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:16.997875 2436 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:17.010342 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:16.970436 6044 raft_consensus.cc:686] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
W20260806 03:52:17.012490 2332 leader_election.cc:334] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.900s (SENT)
I20260806 03:52:17.015545 2332 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:17.020390 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 293) took 1182 ms. Trace:
I20260806 03:52:17.020820 2576 rpcz_store.cc:276] 0806 03:52:15.837698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:15.837815 (+ 117us) service_pool.cc:227] Handling call
0806 03:52:17.020182 (+1182367us) spinlock_profiling.cc:243] Waited 95 ms on lock 0x7b540013bad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:17.020310 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":218427264}
I20260806 03:52:17.023849 2534 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:17.019155 6340 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:17.033926 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:16.962042 6432 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:17.045720 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 231333us
I20260806 03:52:17.050733 2576 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:17.051988 2576 raft_consensus.cc:595] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:17.052567 2576 raft_consensus.cc:603] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:17.054111 2481 raft_consensus.cc:1261] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260806 03:52:17.032641 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 229461us
I20260806 03:52:17.032847 6481 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:17.033885 6087 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:17.060560 6392 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:17.110849 2537 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:17.113166 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 76455.2us
I20260806 03:52:17.114208 2534 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:17.032982 6203 consensus_queue.cc:1046] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
W20260806 03:52:17.119480 2479 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.579s user 0.006s sys 0.042s
W20260806 03:52:17.119858 2479 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.610s user 0.006s sys 0.042s
I20260806 03:52:17.120641 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2072) took 1110 ms. Trace:
I20260806 03:52:17.044486 6489 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:52:17.130496 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 154890us
I20260806 03:52:17.121170 2479 rpcz_store.cc:276] 0806 03:52:16.010473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:16.064071 (+ 53598us) service_pool.cc:227] Handling call
0806 03:52:16.064440 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:16.064689 (+ 249us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:16.336704 (+272015us) log.cc:842] Serialized 14 byte log entry
0806 03:52:17.120205 (+783501us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:17.120267 (+ 62us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:17.120332 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:17.120421 (+ 89us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:17.120561 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33425,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":135691,"raft.run_cpu_time_us":66,"raft.run_wall_time_us":66,"thread_start_us":1054618,"threads_started":3,"wal-append.queue_time_us":505024}
I20260806 03:52:17.068470 6311 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.083673 6361 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.120419 5922 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:17.120807 6455 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:17.120535 6087 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:17.157470 2374 raft_consensus.cc:1261] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:17.129094 6489 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:17.172076 2436 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:17.173080 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 187733us
I20260806 03:52:17.118907 6167 consensus_queue.cc:1035] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20260806 03:52:17.178834 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:17.179807 2478 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
W20260806 03:52:17.155977 6338 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.420s user 0.022s sys 0.087s
I20260806 03:52:17.116755 6392 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.161556 5922 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:17.201313 2576 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260806 03:52:17.169549 6404 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:52:17.187583 6338 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.452s user 0.023s sys 0.087s
I20260806 03:52:17.213209 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.217504 2584 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:17.179549 6489 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:17.213258 2436 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:17.210835 5922 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:17.129606 6044 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:17.327210 2374 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:52:17.196434 6495 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:17.339800 2374 raft_consensus.cc:1261] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:17.334061 6340 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.341835 3586 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:17.202704 5510 consensus_queue.cc:1035] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.136s
I20260806 03:52:17.342940 6392 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.343180 5510 consensus_queue.cc:1225] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer 608e57d72aae496aaf8613622153aa0c log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260806 03:52:17.181257 6087 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:52:17.345490 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 135719us
I20260806 03:52:17.346671 2436 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:17.276078 6404 raft_consensus.cc:686] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:17.373849 2436 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:17.214229 6414 consensus_queue.cc:1035] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.134s
I20260806 03:52:17.325881 6361 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:17.175151 6455 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:17.334307 5682 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:17.381045 2332 leader_election.cc:334] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.158s (SENT)
I20260806 03:52:17.381647 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:17.384171 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 167178us
I20260806 03:52:17.334933 6500 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:17.359526 6392 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:17.423960 2478 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:17.396229 2332 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:17.334187 6386 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.375027 6495 raft_consensus.cc:2741] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election decision vote started in defunct term 6: won
W20260806 03:52:17.441690 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 73332.8us
I20260806 03:52:17.368412 6432 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.378659 6498 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:17.458026 2484 pending_rounds.cc:76] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 1
I20260806 03:52:17.458418 2484 pending_rounds.cc:97] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260806 03:52:17.456404 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:17.458676 2484 raft_consensus.cc:2866] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:17.461835 2476 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 03:52:17.461946 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:17.462751 2372 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:17.400200 6481 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:17.424158 6500 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:17.471823 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 90834.2us
W20260806 03:52:17.476456 2332 leader_election.cc:334] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.203s (SENT)
I20260806 03:52:17.478981 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:17.481508 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:17.482000 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.483328 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.460521 2374 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:17.490440 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:17.468212 6498 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.494490 6498 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.461323 5682 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:17.464964 6432 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:17.507323 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.466631 6167 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:17.518074 6311 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:17.444546 6392 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.527380 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:17.479816 6500 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:17.521100 6498 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:17.499691 6473 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 1.154s user 0.013s sys 0.005s
W20260806 03:52:17.544407 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 68003.5us
W20260806 03:52:17.589318 2332 leader_election.cc:334] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.225s (SENT)
I20260806 03:52:17.589911 2332 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:17.482702 6481 raft_consensus.cc:686] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:17.556660 3590 raft_consensus.cc:2362] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6. Re-sending same reply.
W20260806 03:52:17.501111 6494 log.cc:924] Time spent T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.305s user 0.000s sys 0.005s
I20260806 03:52:17.499594 5682 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:17.567936 2331 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:17.584633 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 61484.4us
I20260806 03:52:17.592657 2581 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:17.594328 2372 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:17.484467 6164 consensus_queue.cc:1046] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.086s
I20260806 03:52:17.605684 6361 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:17.605463 6021 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:17.622305 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 586584us
W20260806 03:52:17.623812 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 55987.6us
I20260806 03:52:17.656201 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.640316 6361 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.488121 6404 consensus_queue.cc:237] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:17.627128 2332 leader_election.cc:334] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.021s (SENT)
I20260806 03:52:17.668093 2332 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:17.668334 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.556528 6167 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:17.609614 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:52:17.652721 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 136864us
I20260806 03:52:17.648702 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:17.661116 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.662350 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.561426 6311 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.669075 2576 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:17.670408 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.671931 2581 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:17.678102 3542 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:17.598735 6498 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.628233 6432 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260806 03:52:17.649060 6021 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.687872 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.649199 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:17.653752 6087 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.658720 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.701701 2577 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
W20260806 03:52:17.658633 6497 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.239s user 0.000s sys 0.001s
I20260806 03:52:17.706430 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:17.647126 5755 consensus_queue.cc:1035] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260806 03:52:17.708493 2480 raft_consensus.cc:3032] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:17.726372 2480 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:17.704859 2574 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
W20260806 03:52:17.712579 6504 log.cc:924] Time spent T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.255s user 0.000s sys 0.001s
I20260806 03:52:17.716351 6386 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:17.723029 6498 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.727316 2480 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.753762 2480 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:17.748860 2574 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "750d734a677a4861bca565b69d29396e"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:17.756425 2574 raft_consensus.cc:595] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:17.766644 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:17.712579 6511 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260806 03:52:17.789149 2480 raft_consensus.cc:1261] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 03:52:17.789747 2480 pending_rounds.cc:76] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 2
I20260806 03:52:17.790030 2480 pending_rounds.cc:97] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:52:17.790325 2480 raft_consensus.cc:2866] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 03:52:17.793943 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 45509us
I20260806 03:52:17.753019 6084 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:17.804390 6084 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:17.797829 6386 raft_consensus.cc:686] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:17.704149 6481 consensus_queue.cc:237] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:17.768505 6087 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:17.807667 6087 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.767818 6432 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:17.809895 6087 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:17.810451 6501 log.cc:924] Time spent T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.315s user 0.000s sys 0.002s
I20260806 03:52:17.818405 6311 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.819707 6483 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:17.820439 6483 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:17.821055 5828 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:17.827607 6361 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.828430 6361 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:17.829092 6361 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:17.829933 3587 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:17.806070 6386 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:17.804945 6084 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:17.753991 6455 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:17.832728 6308 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:17.832933 6164 consensus_queue.cc:1035] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:52:17.834560 2368 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:52:17.843046 2377 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:17.851665 2376 raft_consensus.cc:1261] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:52:17.750474 6508 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:17.859547 2375 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 03:52:17.823845 6483 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:17.874781 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 247676us
I20260806 03:52:17.853338 6361 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.843716 6293 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260806 03:52:17.893210 2332 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:17.903131 2332 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:17.933310 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:17.880533 6392 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.940968 2332 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:17.942410 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:17.945137 2332 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:17.954617 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 265305us
I20260806 03:52:17.957132 2332 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:17.959317 2332 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:17.970311 6167 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:17.971910 6520 log.cc:924] Time spent T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260806 03:52:17.972703 6509 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:17.933655 6430 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:17.900658 6361 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:17.973753 6361 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:17.974848 6283 consensus_meta.cc:287] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.014s user 0.016s sys 0.005s
I20260806 03:52:17.983342 2480 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14fa621f65da481d96d569096ded0a35"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54212
I20260806 03:52:17.983809 2332 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.014904 2480 raft_consensus.cc:595] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Received request to transfer leadership
I20260806 03:52:18.022522 6430 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:18.023629 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1164) took 1104 ms. Trace:
W20260806 03:52:17.962206 6528 log.cc:924] Time spent T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260806 03:52:18.024133 2369 rpcz_store.cc:276] 0806 03:52:16.891951 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:16.895707 (+ 3756us) service_pool.cc:227] Handling call
0806 03:52:16.903786 (+ 8079us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:17.182751 (+278965us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:17.182811 (+ 60us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:17.195913 (+ 13102us) log.cc:842] Serialized 26 byte log entry
0806 03:52:17.457137 (+261224us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:17.457206 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:17.953132 (+495926us) spinlock_profiling.cc:243] Waited 43.1 ms on lock 0x7b5400012cd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:17.953208 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:17.953273 (+ 65us) raft_consensus.cc:1671] finished
0806 03:52:17.953293 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:17.995765 (+ 42472us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5107,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"spinlock_wait_cycles":99223808,"thread_start_us":355300,"threads_started":2,"wal-append.queue_time_us":489609}
I20260806 03:52:18.044143 2377 raft_consensus.cc:1226] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:18.044863 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.044763 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.048739 2373 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:17.951656 6455 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:17.990414 6447 raft_consensus.cc:3032] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:18.052361 2429 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:18.053030 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 232990us
I20260806 03:52:17.963313 6311 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:17.962379 6432 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:18.129097 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:17.982774 5922 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.648s user 0.010s sys 0.003s
I20260806 03:52:17.963574 6508 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:17.977624 6539 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:17.976087 6541 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:17.975932 6361 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.978425 6021 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:17.962900 6087 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:17.976954 6531 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:18.129160 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.039158 6205 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election won for term 7
W20260806 03:52:18.138221 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 85970.2us
I20260806 03:52:18.126942 6447 raft_consensus.cc:729] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:18.129876 6311 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:17.987673 6536 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.156s
I20260806 03:52:18.039513 6509 consensus_queue.cc:1035] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.112s
I20260806 03:52:18.183945 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.189790 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.190798 2375 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:18.125865 6500 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.147444 6533 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:18.177294 6311 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:18.180327 6016 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:18.137358 5922 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.180044 6444 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:18.207178 2436 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.184942 6205 raft_consensus.cc:686] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:18.129242 6430 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.215296 6447 consensus_queue.cc:260] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.143327 6539 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.198925 6533 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:18.147140 6541 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.239460 2482 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:52:18.212242 6016 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.228590 6444 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.247516 6016 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.241469 6551 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:18.168608 6531 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.236526 3542 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:18.234119 5828 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:18.264660 2218 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 44725.5us
I20260806 03:52:18.200399 6360 consensus_queue.cc:1035] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260806 03:52:18.264439 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:18.269408 6205 consensus_queue.cc:237] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.241806 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:18.276154 6414 raft_consensus.cc:3032] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:18.276674 6414 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:18.277287 5922 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.268132 6483 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:18.277590 6414 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.278810 6414 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:18.278959 6308 raft_consensus.cc:979] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:52:18.279455 6308 raft_consensus.cc:1067] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:52:18.249544 6551 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.234436 6084 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:18.283880 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:18.284741 6084 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.263370 5828 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.287616 5828 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:18.264540 6489 consensus_meta.cc:287] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.939s user 0.016s sys 0.000s
I20260806 03:52:18.288494 6489 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.243631 6533 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:18.278190 6483 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:18.289954 3591 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:18.260303 6447 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:18.295918 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 53752.7us
I20260806 03:52:18.276203 3586 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:18.287846 6021 raft_consensus.cc:979] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: : Instructing follower dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:52:18.303596 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.307389 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.316062 2474 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:18.319777 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.323453 2577 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8655b80017d7422082efde215e797b5b"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:18.324218 2577 raft_consensus.cc:595] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:18.306085 2478 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "750d734a677a4861bca565b69d29396e"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54220
I20260806 03:52:18.324869 2478 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:52:18.325245 2478 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:18.326043 6430 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.327178 6541 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.330372 5922 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:18.330984 5922 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.332716 6500 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.333918 6531 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.337268 6489 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:18.339334 6551 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:18.296432 6483 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:18.344434 6539 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.332753 6444 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.353557 5922 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.361128 6483 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.362156 6311 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.392638 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.369316 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:18.378304 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:18.394381 6311 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.396203 3590 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:18.401309 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.402479 2379 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:18.404425 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:18.387028 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.387372 2581 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260806 03:52:18.457830 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.457901 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.458611 2581 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:18.388732 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.459563 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 378) took 1081 ms. Trace:
I20260806 03:52:18.389950 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.390368 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.371670 6533 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.389607 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.391038 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.482110 2577 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:18.475968 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.391182 2478 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.407392 6483 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:18.422803 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.483323 2379 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:18.431190 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:18.442061 2372 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 03:52:18.455201 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:18.379880 5922 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:18.461683 2584 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:18.490720 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:18.490876 2478 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.389607 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:18.465137 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 258061us
I20260806 03:52:18.491894 2583 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:18.465767 6481 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:18.467845 6021 raft_consensus.cc:1067] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Signalling peer dff94100ffc5485f92cbe6a85addf0b2 to start an election
I20260806 03:52:18.470152 3545 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:18.472689 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:18.495441 2369 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 7
I20260806 03:52:18.463806 2575 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:18.477437 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 162514us
I20260806 03:52:18.477932 2331 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.480847 3590 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:18.476006 2374 rpcz_store.cc:276] 0806 03:52:17.379838 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:17.380882 (+ 1044us) service_pool.cc:227] Handling call
0806 03:52:18.459365 (+1078483us) spinlock_profiling.cc:243] Waited 50.8 ms on lock 0x7b54003331d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:18.459469 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":116738432}
I20260806 03:52:18.486048 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:18.488186 2373 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:18.489702 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:18.489423 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:18.491492 2370 raft_consensus.cc:2354] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8.
I20260806 03:52:18.492264 2436 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.492607 2482 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "14fa621f65da481d96d569096ded0a35"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54212
I20260806 03:52:18.492204 2377 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:18.503628 2479 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260806 03:52:18.503913 2436 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.507050 2436 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.507369 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:18.508177 2379 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:18.481496 6500 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:18.505623 2377 raft_consensus.cc:1226] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:18.509122 2436 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.515419 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:18.526486 3590 raft_consensus.cc:2354] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8.
I20260806 03:52:18.527154 2436 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.528877 2584 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:52:18.533245 2436 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:18.540185 2332 leader_election.cc:334] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.587s (SENT)
I20260806 03:52:18.540928 2332 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.541064 2369 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:18.532377 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:18.556170 2436 leader_election.cc:398] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:18.556875 2436 leader_election.cc:401] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 03:52:18.560045 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:18.560575 3545 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:18.563275 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 44215.3us
I20260806 03:52:18.563987 3542 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:18.564257 3591 raft_consensus.cc:1261] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEARNER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:52:18.510492 6430 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:18.567744 2369 raft_consensus.cc:1226] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:18.574687 2377 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "14fa621f65da481d96d569096ded0a35"
dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:40734
I20260806 03:52:18.575564 2377 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:52:18.575968 2377 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:18.573031 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:18.511488 6541 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:18.569226 2436 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:18.514882 6481 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:18.517472 6361 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:18.580399 3545 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:18.584077 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 77142.4us
I20260806 03:52:18.531060 6491 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:18.580942 2482 raft_consensus.cc:595] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Received request to transfer leadership
I20260806 03:52:18.585255 2538 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:18.588601 2436 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.590209 2436 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.543025 6535 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:18.597031 6544 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:18.578985 6536 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:18.598456 6536 raft_consensus.cc:686] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:18.598325 6531 consensus_queue.cc:1046] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:52:18.572669 6087 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:18.599647 6536 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.600589 6087 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:18.601209 6087 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:18.579066 6557 consensus_queue.cc:1046] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.022s
I20260806 03:52:18.579138 6495 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:18.579213 6376 raft_consensus.cc:2781] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:18.587287 6431 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:18.603039 6514 raft_consensus.cc:2741] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 03:52:18.604286 6212 consensus_queue.cc:1035] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:52:18.604471 6414 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:52:18.606231 6212 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:52:18.607019 6541 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:18.600660 6538 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:18.600150 6533 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.587246 6021 raft_consensus.cc:2741] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260806 03:52:18.597271 6556 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:18.610384 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:18.627025 2377 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:18.631140 2369 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:18.516883 6084 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.069s
I20260806 03:52:18.499094 5922 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:18.617699 2332 leader_election.cc:334] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.707s (SENT)
I20260806 03:52:18.656677 2377 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.629308 2436 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:18.642628 2369 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:18.612201 2575 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:18.663517 2429 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:18.544093 6500 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.587154 6491 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:18.648772 2379 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:18.671154 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:18.672293 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.675216 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:18.678956 2481 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
W20260806 03:52:18.676463 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 87810.7us
I20260806 03:52:18.678853 2478 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:18.680107 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.630550 6544 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:18.584412 6419 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.081s
I20260806 03:52:18.706045 5922 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.709022 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.712950 2478 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:18.690286 2429 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:18.681224 3587 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 03:52:18.607043 6340 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260806 03:52:18.690824 6310 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:18.664932 6376 raft_consensus.cc:686] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:18.692320 6556 raft_consensus.cc:686] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:18.693502 6533 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:18.721768 6544 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:18.726976 6544 raft_consensus.cc:686] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:18.720734 6310 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:18.727851 6310 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:18.728093 6556 consensus_queue.cc:237] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.728566 6544 consensus_queue.cc:237] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.696273 6481 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:18.731451 6481 raft_consensus.cc:686] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:18.668099 6021 raft_consensus.cc:3032] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:18.681097 6491 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:18.731963 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.733040 2477 raft_consensus.cc:2379] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:18.733474 6376 consensus_queue.cc:237] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.737270 3586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36484 (request call id 316) took 1225 ms. Trace:
I20260806 03:52:18.701112 6500 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.681929 6203 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:18.713111 2534 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:18.716562 2537 leader_election.cc:398] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8.
I20260806 03:52:18.745193 2537 leader_election.cc:401] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 03:52:18.738365 6021 raft_consensus.cc:729] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:18.689930 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 60748.4us
I20260806 03:52:18.757064 2436 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.720249 2583 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:18.725668 2538 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:18.725627 2578 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:18.762871 2578 raft_consensus.cc:595] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:18.763445 2578 raft_consensus.cc:603] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:18.743343 2377 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 03:52:18.772701 2436 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:18.775990 2436 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.778566 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:18.779644 3590 raft_consensus.cc:2379] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:18.788656 2436 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:18.720252 5828 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260806 03:52:18.800824 2436 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:18.741024 6533 raft_consensus.cc:686] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:18.802034 2534 leader_election.cc:398] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8.
W20260806 03:52:18.805768 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 89370.3us
I20260806 03:52:18.806851 2537 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:18.810304 6574 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:18.797396 6430 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:18.815527 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 49739.2us
I20260806 03:52:18.787710 6500 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:18.780218 6557 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:18.788221 6526 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:18.794453 6535 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:18.796452 6361 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:18.832073 2369 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260806 03:52:18.803021 6541 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:18.812007 6430 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.826849 6500 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.848369 6533 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:18.855674 6564 log.cc:924] Time spent T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260806 03:52:18.800057 6495 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.034s
I20260806 03:52:18.827076 6526 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:18.863480 2480 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:18.737747 3586 rpcz_store.cc:276] 0806 03:52:17.511789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:17.515096 (+ 3307us) service_pool.cc:227] Handling call
0806 03:52:18.736989 (+1221893us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b5400541354. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:18.737180 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":670651648}
I20260806 03:52:18.767047 6203 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.833690 6021 consensus_queue.cc:260] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:18.843473 6557 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:18.873958 6535 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:18.914252 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 113516us
I20260806 03:52:18.902537 5755 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:18.838279 6481 consensus_queue.cc:237] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:18.888182 5829 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8.
I20260806 03:52:18.904532 6495 consensus_queue.cc:1225] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
W20260806 03:52:18.950133 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 85222.7us
I20260806 03:52:18.951805 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:18.952838 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 227250us
I20260806 03:52:18.953902 2538 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:18.927874 6483 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:18.885784 6407 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260806 03:52:18.879647 6311 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:18.930691 6430 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:18.927611 6310 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:18.933014 2235 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:38220 (request call id 128) took 16167 ms (client timeout 15000 ms). Trace:
I20260806 03:52:18.931834 6203 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:18.940471 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.947757 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.957939 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:18.959813 2331 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:18.908601 6526 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:18.962121 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.966506 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:18.980490 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.991856 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:18.992431 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.994179 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.946197 6557 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:18.988346 2479 raft_consensus.cc:2379] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:18.980101 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:19.004343 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:19.005625 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:19.004343 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:19.001325 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:19.006946 2436 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:18.944434 6535 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:19.010492 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.940601 6442 consensus_queue.cc:1046] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.085s
I20260806 03:52:19.025552 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.979478 6021 raft_consensus.cc:3037] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:19.030200 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.995886 6360 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:18.972507 2235 rpcz_store.cc:269] 0806 03:52:02.765777 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:02.773586 (+ 7809us) service_pool.cc:227] Handling call
0806 03:52:18.932916 (+16159330us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:52:13.440751 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:52:13.583806 (+143055us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:52:13.587707 (+ 3901us) write_op.cc:435] Acquired schema lock
0806 03:52:13.587729 (+ 22us) tablet.cc:660] Decoding operations
0806 03:52:13.589240 (+ 1511us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:52:13.589403 (+ 163us) write_op.cc:641] Partition lock acquired for write op
0806 03:52:13.589431 (+ 28us) tablet.cc:683] Acquiring locks for 8 operations
0806 03:52:13.590305 (+ 874us) tablet.cc:699] Row locks acquired
0806 03:52:13.590351 (+ 46us) write_op.cc:260] PREPARE: finished
0806 03:52:13.590605 (+ 254us) write_op.cc:270] Start()
0806 03:52:13.590772 (+ 167us) write_op.cc:276] Timestamp: P: 1785988333590509 usec, L: 0
0806 03:52:13.590815 (+ 43us) op_driver.cc:331] REPLICATION: starting
0806 03:52:13.628222 (+ 37407us) log.cc:842] Serialized 3231 byte log entry
0806 03:52:13.767817 (+139595us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d79b7f5fe 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9ce22 00007f0d7c7465bb
0806 03:52:14.553914 (+786097us) op_driver.cc:444] REPLICATION: finished
0806 03:52:14.972736 (+418822us) write_op.cc:301] APPLY: starting
0806 03:52:15.000148 (+ 27412us) tablet.cc:1365] starting BulkCheckPresence
0806 03:52:15.019276 (+ 19128us) tablet.cc:1368] finished BulkCheckPresence
0806 03:52:15.019303 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:52:15.030310 (+ 11007us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:52:15.030712 (+ 402us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0806 03:52:15.030770 (+ 58us) write_op.cc:312] APPLY: finished
0806 03:52:15.031192 (+ 422us) log.cc:842] Serialized 65 byte log entry
0806 03:52:15.031452 (+ 260us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:52:15.031673 (+ 221us) write_op.cc:454] Released schema lock
0806 03:52:15.031738 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0806 03:52:15.056837 (+ 25099us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:52:15.062471 (+ 5634us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":4251070,"reported_tablets":20,"child_traces":[["op",{"apply.queue_time_us":274015,"apply.run_cpu_time_us":5362,"apply.run_wall_time_us":142491,"num_ops":8,"prepare.queue_time_us":447877,"prepare.run_cpu_time_us":23451,"prepare.run_wall_time_us":375770,"raft.queue_time_us":341532,"raft.run_cpu_time_us":92492,"raft.run_wall_time_us":320180,"replication_time_us":941105,"spinlock_wait_cycles":24451328,"thread_start_us":767428,"threads_started":4,"wal-append.queue_time_us":384869,"wal-append.run_cpu_time_us":35631,"wal-append.run_wall_time_us":2012929}]]}
W20260806 03:52:19.033614 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 156180us
I20260806 03:52:19.008477 6557 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:19.003266 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:19.038709 3591 raft_consensus.cc:1226] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:19.043239 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:18.973256 6414 consensus_queue.cc:1035] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260806 03:52:19.027695 6164 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:18.981460 6311 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:19.031960 6587 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:18.974145 6483 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:19.061892 6311 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:19.069012 6310 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:19.039330 6589 raft_consensus.cc:2741] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260806 03:52:19.073122 6500 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:19.057404 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 103621us
W20260806 03:52:19.113771 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 76090.9us
I20260806 03:52:19.115402 2436 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:19.117278 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:19.127826 2368 raft_consensus.cc:1226] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:19.162657 2484 raft_consensus.cc:1226] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
W20260806 03:52:19.171720 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 184227us
I20260806 03:52:19.130533 6587 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:19.192595 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:19.193305 2374 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 5
W20260806 03:52:19.199156 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 69499.4us
I20260806 03:52:19.201642 2368 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 9. Current term is 10. Ops: []
W20260806 03:52:19.248515 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48377.3us
I20260806 03:52:19.265012 2538 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:19.267371 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:19.274179 6310 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:19.281191 6414 consensus_queue.cc:1046] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:19.283622 6551 raft_consensus.cc:3032] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:19.284037 6551 raft_consensus.cc:729] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:19.285079 6551 consensus_queue.cc:260] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:19.286573 6551 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:52:19.293516 6588 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.143s user 0.000s sys 0.000s
W20260806 03:52:19.295780 6578 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.256s user 0.000s sys 0.001s
W20260806 03:52:19.238346 6568 log.cc:924] Time spent T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
W20260806 03:52:19.315747 2332 leader_election.cc:334] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.986s (SENT)
I20260806 03:52:19.320483 2374 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:19.328850 2534 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:19.297086 6203 raft_consensus.cc:2781] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
W20260806 03:52:19.334444 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 216566us
I20260806 03:52:19.194984 6491 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:19.339587 2436 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:19.336937 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 111811us
W20260806 03:52:19.328058 6580 log.cc:924] Time spent T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.319s user 0.001s sys 0.000s
I20260806 03:52:19.270905 6431 consensus_queue.cc:1046] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.069s
I20260806 03:52:19.356484 6361 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:19.406141 6556 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:19.411033 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 73334.5us
I20260806 03:52:19.414726 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:19.418550 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 48114.4us
W20260806 03:52:19.404211 6087 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.802s user 0.013s sys 0.005s
I20260806 03:52:19.420403 2332 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:19.420953 2374 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260806 03:52:19.392380 6203 raft_consensus.cc:686] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:19.421044 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:19.424047 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:19.429184 6599 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:19.408277 6483 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:19.430383 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:19.433863 2332 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:19.439963 2332 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:19.441617 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:19.443697 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:19.446386 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:19.418190 6361 raft_consensus.cc:686] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:19.447659 2481 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:19.404203 6419 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260806 03:52:19.466987 2578 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:19.467998 2578 raft_consensus.cc:595] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:19.468420 2578 raft_consensus.cc:603] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:19.432492 3587 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEARNER]: Advancing to term 6
I20260806 03:52:19.443905 6084 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:19.451952 6596 consensus_queue.cc:1046] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260806 03:52:19.445598 6589 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:19.426110 6556 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:19.462356 2538 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:19.450117 2481 raft_consensus.cc:1226] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:19.474371 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:19.479303 6481 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:19.485460 2480 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 1.
I20260806 03:52:19.448180 6599 raft_consensus.cc:686] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:19.429287 6087 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:19.505391 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:19.505779 6615 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:19.506850 6602 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 03:52:19.508133 6361 consensus_queue.cc:237] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:19.512259 6203 consensus_queue.cc:237] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:19.513092 6506 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:19.513952 6506 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:19.516274 6594 log.cc:924] Time spent T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.307s user 0.000s sys 0.001s
I20260806 03:52:19.518245 6603 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:19.457041 5682 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260806 03:52:19.525741 6556 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:19.530028 6087 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:19.533208 2369 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 6: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:52:19.545543 6535 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.514s user 0.011s sys 0.002s
W20260806 03:52:19.562844 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 80102.6us
I20260806 03:52:19.523880 6373 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:19.566501 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:19.589923 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:19.501055 6481 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:19.623322 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 122483us
I20260806 03:52:19.687048 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:19.694253 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 55203.2us
I20260806 03:52:19.596961 6491 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:19.586874 6535 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:19.656138 6310 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:19.723706 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 47047.3us
I20260806 03:52:19.808676 2483 raft_consensus.cc:2354] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9.
I20260806 03:52:19.821655 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260806 03:52:19.823262 6593 log.cc:924] Time spent T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.285s user 0.002s sys 0.000s
W20260806 03:52:19.822844 6618 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.172s user 0.000s sys 0.000s
I20260806 03:52:19.839226 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:52:19.859931 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 129601us
W20260806 03:52:19.921902 6526 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.875s user 0.015s sys 0.002s
I20260806 03:52:20.015786 6615 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:20.016731 6615 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:20.021683 6628 log.cc:924] Time spent T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260806 03:52:20.005055 6308 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:20.005049 6602 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:20.024924 6308 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:20.026510 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:52:19.694638 6599 consensus_queue.cc:237] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:20.028931 6535 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:20.031993 6308 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:20.034859 6619 log.cc:924] Time spent T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.200s user 0.000s sys 0.002s
I20260806 03:52:19.835655 3584 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:52:20.043314 6310 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:20.025014 6602 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:20.068693 6481 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:20.055109 2572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2320) took 1099 ms. Trace:
W20260806 03:52:20.093042 2583 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.006s sys 0.001s
W20260806 03:52:20.178454 2583 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.599s user 0.006s sys 0.001s
I20260806 03:52:20.178169 2572 rpcz_store.cc:276] 0806 03:52:18.955534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.955734 (+ 200us) service_pool.cc:227] Handling call
0806 03:52:18.956146 (+ 412us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:19.217526 (+261380us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:19.217572 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:19.268315 (+ 50743us) log.cc:842] Serialized 26 byte log entry
0806 03:52:19.496939 (+228624us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:19.496985 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:19.497081 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:20.054760 (+557679us) raft_consensus.cc:1671] finished
0806 03:52:20.054789 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:20.055010 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51585,"raft.queue_time_us":29712,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":13734,"thread_start_us":423163,"threads_started":2,"wal-append.queue_time_us":368144}
I20260806 03:52:20.051182 6506 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:20.189083 3587 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.695s user 0.014s sys 0.000s
I20260806 03:52:20.301362 3587 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEARNER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:20.302700 6636 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:20.014596 6526 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:20.242753 6310 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:20.454209 6602 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:20.455107 6021 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:20.455186 6602 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:20.456139 6021 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:20.303680 6636 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:20.509202 2584 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.786s user 0.009s sys 0.009s
W20260806 03:52:20.517333 2584 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.794s user 0.009s sys 0.009s
W20260806 03:52:20.481539 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.000s sys 0.014s
W20260806 03:52:20.628682 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.709s user 0.000s sys 0.015s
W20260806 03:52:20.680478 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 1.23676e+06us
W20260806 03:52:20.680943 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 761642us
I20260806 03:52:20.682607 2332 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:20.684850 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 1.04741e+06us
W20260806 03:52:20.685631 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 982034us
I20260806 03:52:20.679630 2576 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:52:20.702442 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2326) took 1076 ms. Trace:
I20260806 03:52:20.702981 2576 rpcz_store.cc:276] 0806 03:52:19.625940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.626073 (+ 133us) service_pool.cc:227] Handling call
0806 03:52:20.679077 (+1053004us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:20.701980 (+ 22903us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:20.702335 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:20.684285 2332 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:20.748086 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2151) took 1021 ms. Trace:
I20260806 03:52:20.748865 2480 rpcz_store.cc:276] 0806 03:52:19.726836 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.727139 (+ 303us) service_pool.cc:227] Handling call
0806 03:52:20.747244 (+1020105us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:20.747576 (+ 332us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:20.747616 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:20.747674 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:20.747732 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:20.747804 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:20.747965 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:20.754200 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:20.728345 6557 consensus_meta.cc:287] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.675s user 0.016s sys 0.013s
I20260806 03:52:20.759152 2332 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:20.762115 2474 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.165s user 0.003s sys 0.023s
W20260806 03:52:20.762555 2474 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.165s user 0.003s sys 0.023s
I20260806 03:52:20.483899 6602 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:20.787529 2429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.959s user 0.007s sys 0.014s
I20260806 03:52:20.786238 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:20.788332 2429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.024s user 0.007s sys 0.014s
I20260806 03:52:20.801820 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:20.814837 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:20.816431 2478 raft_consensus.cc:3032] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:20.816877 2478 raft_consensus.cc:729] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:20.820112 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.099s user 0.030s sys 0.045s
I20260806 03:52:20.821578 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:20.821832 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1092) took 1096 ms. Trace:
W20260806 03:52:20.823465 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.103s user 0.031s sys 0.045s
I20260806 03:52:20.822656 2577 rpcz_store.cc:276] 0806 03:52:19.725946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:20.774126 (+1048180us) service_pool.cc:227] Handling call
0806 03:52:20.821749 (+ 47623us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:20.782397 6632 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:20.828351 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:20.818805 2478 consensus_queue.cc:260] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:20.833603 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:20.815979 6021 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:20.815800 6631 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:20.847069 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:20.822892 6636 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:20.825888 6526 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:20.785122 6557 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:20.874967 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2329) took 1231 ms. Trace:
I20260806 03:52:20.880446 2584 rpcz_store.cc:276] 0806 03:52:19.643421 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.644752 (+ 1331us) service_pool.cc:227] Handling call
0806 03:52:19.693047 (+ 48295us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:19.693346 (+ 299us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:52:20.517945 (+824599us) log.cc:842] Serialized 14 byte log entry
0806 03:52:20.874411 (+356466us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:20.874491 (+ 80us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:20.874560 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:20.874662 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:20.874868 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9446,"prepare.queue_time_us":344934,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"thread_start_us":1141205,"threads_started":2}
I20260806 03:52:20.882869 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:20.883978 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1093) took 1077 ms. Trace:
I20260806 03:52:20.884402 2584 rpcz_store.cc:276] 0806 03:52:19.806795 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:20.882417 (+1075622us) service_pool.cc:227] Handling call
0806 03:52:20.883893 (+ 1476us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:20.885533 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:20.886961 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:20.889084 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:20.889724 2577 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:20.838071 6648 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:20.888512 2327 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.010s sys 0.012s
W20260806 03:52:20.907776 2327 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.648s user 0.010s sys 0.012s
I20260806 03:52:20.910024 6021 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:20.900115 2584 raft_consensus.cc:2396] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 03:52:20.944504 2476 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: []
W20260806 03:52:20.912837 2433 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.010s sys 0.016s
W20260806 03:52:20.945317 2433 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.867s user 0.011s sys 0.016s
I20260806 03:52:20.927587 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2217) took 2094 ms. Trace:
W20260806 03:52:20.947280 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.029s user 0.014s sys 0.042s
W20260806 03:52:20.947690 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.064s user 0.015s sys 0.043s
W20260806 03:52:20.912678 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.013s sys 0.005s
I20260806 03:52:20.879808 6636 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:20.946672 2372 rpcz_store.cc:276] 0806 03:52:18.833534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.833754 (+ 220us) service_pool.cc:227] Handling call
0806 03:52:18.834131 (+ 377us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:18.909195 (+ 75064us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:18.909246 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:18.909305 (+ 59us) raft_consensus.cc:1631] Marking committed up to 1
0806 03:52:18.909350 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:20.927210 (+2017860us) spinlock_profiling.cc:243] Waited 74.8 ms on lock 0x7b54004385d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:20.927271 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:20.927479 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":172021120}
W20260806 03:52:20.912391 2329 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.000s sys 0.023s
W20260806 03:52:20.956825 2329 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.878s user 0.001s sys 0.024s
I20260806 03:52:20.928293 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2135) took 2203 ms. Trace:
W20260806 03:52:20.951578 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.713s user 0.014s sys 0.006s
W20260806 03:52:20.937337 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.001s sys 0.044s
W20260806 03:52:20.959394 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.522s user 0.001s sys 0.045s
I20260806 03:52:20.934099 2577 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:20.960393 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2331) took 1255 ms. Trace:
I20260806 03:52:20.960975 2577 rpcz_store.cc:276] 0806 03:52:19.704780 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:20.848991 (+1144211us) service_pool.cc:227] Handling call
0806 03:52:20.960323 (+111332us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:20.957520 2484 rpcz_store.cc:276] 0806 03:52:18.725193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.725329 (+ 136us) service_pool.cc:227] Handling call
0806 03:52:18.729288 (+ 3959us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:19.163430 (+434142us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:20.928005 (+1764575us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b54002ffdd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:20.928226 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":995721472}
W20260806 03:52:20.899202 2481 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.018s sys 0.025s
W20260806 03:52:20.963608 2481 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.085s user 0.019s sys 0.026s
I20260806 03:52:20.900168 2576 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:20.981982 2581 raft_consensus.cc:1261] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:20.896505 6652 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:20.904299 6551 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.858s user 0.011s sys 0.019s
I20260806 03:52:20.955945 3587 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260806 03:52:21.234465 6649 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:21.187888 6643 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:21.222078 6636 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:21.285320 6430 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:21.349221 6430 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:21.052564 6551 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.006s user 0.011s sys 0.020s
I20260806 03:52:20.866762 6631 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:21.353209 6642 log.cc:924] Time spent T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.464s user 0.002s sys 0.000s
I20260806 03:52:21.355440 6631 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:20.948653 6164 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:21.356694 6636 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:20.916805 6021 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:21.360153 6654 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:21.360783 6021 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:21.361724 6021 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:21.361033 6654 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:21.362406 2438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.770s user 0.026s sys 0.033s
I20260806 03:52:21.375653 6021 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:21.376616 2438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.195s user 0.026s sys 0.034s
W20260806 03:52:21.329464 2480 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.009s sys 0.039s
W20260806 03:52:21.379189 2480 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s user 0.009s sys 0.040s
I20260806 03:52:21.291343 6602 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:21.329909 3545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.578s user 0.013s sys 0.045s
I20260806 03:52:21.301196 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2212) took 2469 ms. Trace:
I20260806 03:52:20.944597 6648 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:20.907333 6526 raft_consensus.cc:2932] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Committing config change with OpId 7.6: config changed from index 3 to 6, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:21.394411 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54212 (request call id 457) took 2924 ms. Trace:
I20260806 03:52:21.395015 2482 rpcz_store.cc:276] 0806 03:52:18.470095 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.472430 (+ 2335us) service_pool.cc:227] Handling call
0806 03:52:20.815807 (+2343377us) spinlock_profiling.cc:243] Waited 97.6 ms on lock 0x7b5400432720. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7ca5adc0 00007f0d7cabf9c2 00007f0d81a0981a 00007f0d7b0fcd04 00007f0d7b0fcc92 00007f0d7b0fcbc1 00007f0d7b0fcb3c 00007f0d7b0fbdf2 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:21.394139 (+578332us) spinlock_profiling.cc:243] Waited 77.3 ms on lock 0x7b54002a08d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cabfbc3 00007f0d81a0981a 00007f0d7b0fcd04 00007f0d7b0fcc92 00007f0d7b0fcbc1 00007f0d7b0fcb3c 00007f0d7b0fbdf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:21.394287 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":402448640}
I20260806 03:52:21.221180 6652 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:21.381310 6646 log.cc:924] Time spent T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
W20260806 03:52:21.330209 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.016s sys 0.009s
W20260806 03:52:21.350000 6659 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.161s user 0.000s sys 0.001s
I20260806 03:52:21.357511 6164 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:20.835026 6632 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:20.910281 6651 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:21.328675 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.021s sys 0.028s
W20260806 03:52:21.427857 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.643s user 0.021s sys 0.029s
W20260806 03:52:21.382015 3545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.635s user 0.013s sys 0.047s
I20260806 03:52:21.382263 2377 rpcz_store.cc:276] 0806 03:52:18.823228 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.825222 (+ 1994us) service_pool.cc:227] Handling call
0806 03:52:18.825687 (+ 465us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:18.909574 (+ 83887us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:18.909624 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:18.910282 (+ 658us) log.cc:842] Serialized 26 byte log entry
0806 03:52:19.052148 (+141866us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:19.052210 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:21.286748 (+2234538us) spinlock_profiling.cc:243] Waited 83.3 ms on lock 0x7b54001077d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:21.286806 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:21.286887 (+ 81us) raft_consensus.cc:1671] finished
0806 03:52:21.286903 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:21.292253 (+ 5350us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3007,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":6976,"spinlock_wait_cycles":191675776,"thread_start_us":122779,"threads_started":1,"wal-append.queue_time_us":220618,"wal-append.run_cpu_time_us":5312,"wal-append.run_wall_time_us":1879036}
I20260806 03:52:21.378279 6021 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:21.410202 6556 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:21.433974 6556 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:21.411301 6648 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:21.396553 6557 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:21.455358 6631 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:21.456341 6631 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:21.462965 6205 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:21.463951 6631 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:21.464745 6430 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:21.466006 6430 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:21.472054 6556 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:21.473760 6632 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:21.378849 6649 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:21.477955 6649 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:21.478854 6650 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:21.479351 6525 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:21.399430 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.638s user 0.016s sys 0.010s
I20260806 03:52:21.414915 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2221) took 2304 ms. Trace:
W20260806 03:52:21.481299 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 754845us
I20260806 03:52:21.482343 2332 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:21.411445 6657 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:21.445029 2369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 412) took 1940 ms (client timeout 1800 ms). Trace:
I20260806 03:52:21.447057 6652 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:21.487648 2369 rpcz_store.cc:269] 0806 03:52:19.504884 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.505047 (+ 163us) service_pool.cc:227] Handling call
0806 03:52:21.444708 (+1939661us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b54003331d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:21.444926 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61129472}
I20260806 03:52:21.447348 6376 raft_consensus.cc:3032] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:52:21.451319 6651 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:21.503437 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1102) took 1430 ms. Trace:
I20260806 03:52:21.431046 6483 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:21.479557 6650 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:21.499424 6607 log.cc:924] Time spent T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 1.761s user 0.004s sys 0.024s
I20260806 03:52:21.381402 6643 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:21.503295 6376 raft_consensus.cc:729] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:21.480903 2370 rpcz_store.cc:276] 0806 03:52:19.110203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.113360 (+ 3157us) service_pool.cc:227] Handling call
0806 03:52:21.414042 (+2300682us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:52:21.414305 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:21.414522 (+ 217us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:21.414550 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:21.414590 (+ 40us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:21.414617 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:21.414680 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:21.414835 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3336064}
I20260806 03:52:21.466797 6430 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:21.504725 2581 rpcz_store.cc:276] 0806 03:52:20.072796 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:20.919250 (+846454us) service_pool.cc:227] Handling call
0806 03:52:20.981453 (+ 62203us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:20.982470 (+ 1017us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:21.503192 (+520722us) spinlock_profiling.cc:243] Waited 61.7 ms on lock 0x7b540028dcd0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:21.503362 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":142014336}
W20260806 03:52:21.511715 2368 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.710s user 0.020s sys 0.020s
I20260806 03:52:21.459510 6422 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:21.516551 2332 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:21.522327 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 326) took 1952 ms. Trace:
I20260806 03:52:21.537482 2579 rpcz_store.cc:276] 0806 03:52:19.569599 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.572196 (+ 2597us) service_pool.cc:227] Handling call
0806 03:52:19.572595 (+ 399us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:19.593197 (+ 20602us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:19.593242 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:19.593301 (+ 59us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:19.593347 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:21.522062 (+1928715us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x7b54004457d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:21.522115 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:21.522261 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46595840}
I20260806 03:52:21.539651 2332 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:21.545615 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:21.547355 2372 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:21.511531 6654 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:21.503682 6651 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:21.534142 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:21.534945 2481 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.024s sys 0.035s
I20260806 03:52:21.463812 6205 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:21.555435 2370 raft_consensus.cc:1261] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260806 03:52:21.559247 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.019s sys 0.020s
W20260806 03:52:21.570302 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.606s user 0.020s sys 0.020s
I20260806 03:52:21.559659 2372 raft_consensus.cc:1226] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 8. Ops: []
W20260806 03:52:21.566366 2481 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.024s sys 0.035s
I20260806 03:52:21.505091 6648 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:21.572263 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2152) took 1845 ms. Trace:
I20260806 03:52:21.605719 2481 rpcz_store.cc:276] 0806 03:52:19.727178 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.728673 (+ 1495us) service_pool.cc:227] Handling call
0806 03:52:19.729060 (+ 387us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:19.729307 (+ 247us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:52:20.964067 (+1234760us) log.cc:842] Serialized 14 byte log entry
0806 03:52:21.571807 (+607740us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:21.571869 (+ 62us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:21.571946 (+ 77us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:21.572053 (+ 107us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:21.572188 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":193275,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":115,"thread_start_us":1675281,"threads_started":2}
I20260806 03:52:21.577790 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:21.566483 2482 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:21.611073 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:21.611966 2481 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:21.614097 2481 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260806 03:52:21.614946 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2156) took 1886 ms. Trace:
I20260806 03:52:21.615438 2481 rpcz_store.cc:276] 0806 03:52:19.728404 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:21.613270 (+1884866us) service_pool.cc:227] Handling call
0806 03:52:21.613730 (+ 460us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:21.614637 (+ 907us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:21.614882 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:21.532401 2368 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.730s user 0.021s sys 0.020s
I20260806 03:52:21.479894 6525 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:21.608568 2373 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:21.513963 6483 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:21.985805 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.000s user 0.017s sys 0.016s
I20260806 03:52:21.761565 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2249) took 2813 ms. Trace:
I20260806 03:52:22.003753 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:21.648124 3542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.961s user 0.021s sys 0.033s
I20260806 03:52:21.823943 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:22.014914 2372 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:21.718829 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:21.971350 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2261) took 3015 ms. Trace:
I20260806 03:52:21.818219 3586 consensus_queue.cc:237] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.018545 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:21.638882 2377 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260806 03:52:22.015091 3542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.328s user 0.024s sys 0.041s
I20260806 03:52:22.028004 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:22.029201 2372 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
W20260806 03:52:22.014315 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.028s user 0.017s sys 0.017s
W20260806 03:52:22.030337 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 1.04456e+06us
I20260806 03:52:22.027719 2476 rpcz_store.cc:276] 0806 03:52:18.948535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.950936 (+ 2401us) service_pool.cc:227] Handling call
0806 03:52:18.986193 (+ 35257us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:20.950981 (+1964788us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:21.761211 (+810230us) spinlock_profiling.cc:243] Waited 1.96 s on lock 0x7b54002b1954. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:21.761471 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4503180160}
I20260806 03:52:22.017513 2477 rpcz_store.cc:276] 0806 03:52:18.956126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.035077 (+ 78951us) service_pool.cc:227] Handling call
0806 03:52:19.035442 (+ 365us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:19.039220 (+ 3778us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:19.039250 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:19.043311 (+ 4061us) log.cc:842] Serialized 26 byte log entry
0806 03:52:19.297660 (+254349us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:19.297703 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:19.297777 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:21.496434 (+2198657us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:21.828718 (+332284us) raft_consensus.cc:1671] finished
0806 03:52:21.828752 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:21.971205 (+142453us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":265,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":6595,"spinlock_wait_cycles":12946816,"thread_start_us":191418,"threads_started":1,"wal-append.queue_time_us":535883}
I20260806 03:52:21.720857 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2155) took 1992 ms. Trace:
I20260806 03:52:21.758725 5510 raft_consensus.cc:979] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:52:22.046651 2482 rpcz_store.cc:276] 0806 03:52:19.728082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:21.610266 (+1882184us) service_pool.cc:227] Handling call
0806 03:52:21.720170 (+109904us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:21.720440 (+ 270us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:21.720478 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:21.720541 (+ 63us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:21.720589 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:21.720653 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:21.720779 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:21.718890 6376 consensus_queue.cc:260] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.051556 2478 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:52:22.052968 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:22.054149 2379 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 8: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:22.057694 2538 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:22.059391 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:21.511435 6657 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:22.061228 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.980s user 0.014s sys 0.022s
W20260806 03:52:22.061976 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.208s user 0.023s sys 0.092s
W20260806 03:52:22.071341 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.217s user 0.023s sys 0.092s
I20260806 03:52:22.061653 2374 raft_consensus.cc:2354] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6.
I20260806 03:52:21.564870 6422 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:22.086621 2438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.709s user 0.013s sys 0.021s
W20260806 03:52:22.087145 2438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.710s user 0.013s sys 0.021s
W20260806 03:52:22.068959 2484 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s user 0.015s sys 0.026s
W20260806 03:52:22.091672 2484 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.851s user 0.015s sys 0.026s
W20260806 03:52:22.082144 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.001s user 0.014s sys 0.022s
I20260806 03:52:22.054697 6311 consensus_queue.cc:1046] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260806 03:52:22.064574 5884 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:22.104081 5884 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:22.104846 5884 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:22.104349 6311 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.108417 6311 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.109059 6205 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:22.109778 6205 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:22.110277 6205 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.111318 6311 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:22.072759 5828 raft_consensus.cc:1067] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:52:22.113166 6651 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:22.113708 6311 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:22.114316 6311 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.114835 6525 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:22.116374 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 51364.5us
I20260806 03:52:21.817886 6666 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:52:22.126575 2583 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.004s sys 0.039s
W20260806 03:52:22.154224 2583 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.926s user 0.004s sys 0.039s
I20260806 03:52:22.082967 2372 raft_consensus.cc:2932] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 3 to 6, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:22.132061 5829 consensus_queue.cc:1046] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:22.068840 6582 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.060988 6431 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:52:22.115767 2429 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.305s user 0.020s sys 0.057s
I20260806 03:52:22.139577 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:22.157743 2579 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 6: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:22.141434 6533 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.142191 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2140) took 2746 ms. Trace:
I20260806 03:52:22.142346 6650 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:22.143049 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:22.092475 6622 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.120047 6495 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.146231 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:22.167529 2572 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:22.137550 6669 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:22.120235 6652 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:22.150817 6643 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:22.085836 6648 raft_consensus.cc:2781] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 03:52:22.114843 6651 raft_consensus.cc:3032] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Stepping down as leader of term 7
W20260806 03:52:22.157254 2429 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.367s user 0.021s sys 0.057s
W20260806 03:52:22.136227 6164 consensus_meta.cc:287] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.698s user 0.008s sys 0.007s
I20260806 03:52:22.161679 2474 rpcz_store.cc:276] 0806 03:52:19.395387 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.395912 (+ 525us) service_pool.cc:227] Handling call
0806 03:52:19.419573 (+ 23661us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:19.527749 (+108176us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:19.592765 (+ 65016us) log.cc:842] Serialized 14 byte log entry
0806 03:52:20.762993 (+1170228us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:20.763052 (+ 59us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:20.763104 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:22.141838 (+1378734us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54001de2d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:22.141898 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:22.142108 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56191,"prepare.queue_time_us":165846,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":99978,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":15311,"spinlock_wait_cycles":249258112,"thread_start_us":1229659,"threads_started":2,"wal-append.queue_time_us":1358691}
I20260806 03:52:22.158731 5829 consensus_queue.cc:1225] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer 608e57d72aae496aaf8613622153aa0c log is divergent from this leader: its last log entry 7.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260806 03:52:22.157156 6666 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.173020 6495 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:22.181074 3545 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.009s sys 0.018s
W20260806 03:52:22.186372 2327 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.135s user 0.022s sys 0.026s
W20260806 03:52:22.187696 2327 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.199s user 0.023s sys 0.026s
W20260806 03:52:22.185910 3545 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.575s user 0.009s sys 0.019s
W20260806 03:52:22.194125 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 583176us
W20260806 03:52:22.198443 3545 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.3:46407 timed out after 1.972s (SENT) [suppressed 5 similar messages]
I20260806 03:52:22.175493 6648 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.182101 6643 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:22.176177 6651 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:22.164245 6431 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.204744 6525 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:22.187409 6666 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:22.226025 2368 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.005s sys 0.015s
I20260806 03:52:22.192427 6657 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:22.195736 6430 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:22.240928 2578 raft_consensus.cc:1226] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:22.211889 6648 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:52:22.165221 6533 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:22.243631 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.024s sys 0.037s
W20260806 03:52:22.253059 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.815s user 0.024s sys 0.037s
I20260806 03:52:22.246649 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:22.247906 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:22.179242 6652 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:22.244417 2368 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s user 0.005s sys 0.015s
I20260806 03:52:22.256758 3590 raft_consensus.cc:2354] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6.
I20260806 03:52:22.187002 6164 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.170094 6622 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.176236 6669 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.164436 6582 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.236229 6643 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.260896 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:22.293865 6687 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:22.294641 6683 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:22.294844 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:22.259990 6673 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:22.296463 6422 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:22.297470 6671 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.274101 3584 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:22.273584 6311 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:22.298466 6622 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:22.244899 6525 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.299332 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:22.281109 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:22.301005 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2154) took 2573 ms. Trace:
I20260806 03:52:22.347425 2484 rpcz_store.cc:276] 0806 03:52:19.727785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:20.962033 (+1234248us) service_pool.cc:227] Handling call
0806 03:52:20.962466 (+ 433us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:20.988726 (+ 26260us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:20.988773 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:21.052196 (+ 63423us) log.cc:842] Serialized 26 byte log entry
0806 03:52:22.097161 (+1044965us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:22.097206 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:22.097292 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:22.300720 (+203428us) raft_consensus.cc:1671] finished
0806 03:52:22.300748 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:22.300931 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25806,"raft.queue_time_us":443272,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":179,"thread_start_us":856643,"threads_started":1,"wal-append.queue_time_us":1012684}
I20260806 03:52:22.301430 6203 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:22.301573 6495 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:22.302805 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 320) took 2738 ms. Trace:
I20260806 03:52:22.305059 2478 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:22.279560 6483 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:22.321534 2581 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:22.322815 6028 raft_consensus.cc:3032] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:22.316653 6687 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.327455 6544 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:22.335168 2480 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.955s user 0.025s sys 0.032s
W20260806 03:52:22.371995 2480 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.992s user 0.026s sys 0.033s
I20260806 03:52:22.298473 6533 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:22.376288 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.005s sys 0.020s
W20260806 03:52:22.376782 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.679s user 0.005s sys 0.020s
I20260806 03:52:22.376899 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:22.373121 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2153) took 2645 ms. Trace:
W20260806 03:52:22.359579 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.789s user 0.016s sys 0.043s
I20260806 03:52:22.352727 2583 rpcz_store.cc:276] 0806 03:52:19.563988 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.564531 (+ 543us) service_pool.cc:227] Handling call
0806 03:52:19.564890 (+ 359us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:20.179325 (+614435us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:20.179386 (+ 61us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:20.936780 (+757394us) spinlock_profiling.cc:243] Waited 93.3 ms on lock 0x7b2c001765a8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:52:20.947008 (+ 10228us) log.cc:842] Serialized 26 byte log entry
0806 03:52:22.154730 (+1207722us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:22.154771 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:22.154859 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:22.302402 (+147543us) raft_consensus.cc:1671] finished
0806 03:52:22.302437 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:22.302714 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16715,"raft.queue_time_us":86356,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":45538,"spinlock_wait_cycles":214491904,"thread_start_us":1525760,"threads_started":2,"wal-append.queue_time_us":1276910}
I20260806 03:52:22.277400 6651 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:22.388736 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.818s user 0.016s sys 0.043s
W20260806 03:52:22.394552 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 114076us
I20260806 03:52:22.328943 6671 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:22.416764 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 141265us
I20260806 03:52:22.422991 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:22.329562 6683 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.357889 6203 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.329293 6673 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:22.417333 2332 connection.cc:232] RPC call timeout handler was delayed by -1.6059342930093408s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1565s for a response
W20260806 03:52:23.109830 2331 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.660s user 0.001s sys 0.000s
W20260806 03:52:23.147529 2331 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.698s user 0.003s sys 0.002s
I20260806 03:52:22.443610 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:23.149359 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 721481us
W20260806 03:52:23.134047 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.506s user 0.000s sys 0.022s
W20260806 03:52:23.150419 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.522s user 0.000s sys 0.030s
W20260806 03:52:23.108824 2372 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s user 0.009s sys 0.007s
I20260806 03:52:22.386834 2480 rpcz_store.cc:276] 0806 03:52:19.727472 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:20.750605 (+1023133us) service_pool.cc:227] Handling call
0806 03:52:20.784560 (+ 33955us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:20.784871 (+ 311us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:20.812877 (+ 28006us) log.cc:842] Serialized 14 byte log entry
0806 03:52:22.372555 (+1559678us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:22.372644 (+ 89us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:22.372725 (+ 81us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:22.372834 (+ 109us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:22.373032 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":248818,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":54940,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":235,"thread_start_us":1558653,"threads_started":2,"wal-append.queue_time_us":1009665}
I20260806 03:52:22.431659 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:22.487340 2429 connection.cc:232] RPC call timeout handler was delayed by -1.4423316272441298s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
W20260806 03:52:23.152613 2372 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s user 0.010s sys 0.012s
I20260806 03:52:22.646682 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2328) took 3019 ms. Trace:
I20260806 03:52:23.192402 2578 rpcz_store.cc:276] 0806 03:52:19.626829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.626972 (+ 143us) service_pool.cc:227] Handling call
0806 03:52:19.627376 (+ 404us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:22.247505 (+2620129us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:22.646384 (+398879us) spinlock_profiling.cc:243] Waited 2.61 s on lock 0x7b540004b854. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:22.646605 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6010176256}
I20260806 03:52:23.194470 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:23.195788 3586 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.002s sys 0.009s
W20260806 03:52:23.196511 3586 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.700s user 0.002s sys 0.009s
W20260806 03:52:22.966508 6361 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 3.142s user 0.020s sys 0.043s
W20260806 03:52:23.197688 6361 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 3.373s user 0.022s sys 0.045s
I20260806 03:52:22.394606 6311 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:23.199257 6361 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:22.666764 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 179247us
I20260806 03:52:23.199690 6311 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.269035 6361 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:22.420480 6430 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:23.271260 6430 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:22.643303 5884 consensus_meta.cc:287] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.537s user 0.013s sys 0.012s
I20260806 03:52:23.273362 5884 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:23.127636 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.564s user 0.003s sys 0.002s
I20260806 03:52:22.395195 6682 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:23.326056 6682 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:23.327926 2575 raft_consensus.cc:2354] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 8. Current term is 9.
I20260806 03:52:22.418802 6657 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:23.329331 6587 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:23.329758 6538 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:23.330506 6587 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.330946 6538 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:22.362418 6028 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:22.647564 6582 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.353027 6587 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.354032 6587 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:23.354249 6028 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.396135 6544 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.358109 2572 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:52:22.882118 6648 consensus_meta.cc:287] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.622s user 0.014s sys 0.001s
I20260806 03:52:22.746896 6666 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:22.646940 6526 consensus_meta.cc:287] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.257s user 0.010s sys 0.008s
I20260806 03:52:23.386533 2370 raft_consensus.cc:1226] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:23.386023 6648 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:22.386307 6674 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:23.361409 6683 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:23.355183 6587 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.401762 5884 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:22.768335 6688 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:23.403867 6587 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:23.386138 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.823s user 0.007s sys 0.004s
I20260806 03:52:23.389382 6674 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:23.386027 6203 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:22.384418 6495 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:22.742352 6643 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:22.542239 6651 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:23.036545 6539 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.010s sys 0.013s
I20260806 03:52:22.489593 6650 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:23.397404 6671 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:22.408833 6533 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:22.647477 6431 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.282040 6430 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.389873 6648 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.403797 6688 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:23.580289 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 225559us
I20260806 03:52:22.384189 6622 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:23.593291 6539 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.092s user 0.011s sys 0.014s
W20260806 03:52:23.593878 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 1.03073e+06us
I20260806 03:52:23.565702 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2249) took 1834 ms. Trace:
I20260806 03:52:23.641768 6447 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:22.646230 6692 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260806 03:52:23.329324 6657 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.668830 6447 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:23.670017 6666 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:23.676041 6483 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.962s user 0.003s sys 0.027s
I20260806 03:52:22.764955 6525 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:23.682950 2537 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 1.041s user 0.000s sys 0.013s
I20260806 03:52:23.559433 6587 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:22.781042 6669 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:23.643421 6311 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:23.660224 3545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.087s user 0.000s sys 0.012s
I20260806 03:52:23.644732 2372 rpcz_store.cc:276] 0806 03:52:21.731099 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.067099 (+336000us) service_pool.cc:227] Handling call
0806 03:52:22.082424 (+ 15325us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:22.082788 (+ 364us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:52:22.374062 (+291274us) log.cc:842] Serialized 14 byte log entry
0806 03:52:23.564756 (+1190694us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:23.564896 (+ 140us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:23.565025 (+ 129us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:23.565198 (+ 173us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:23.565562 (+ 364us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":78066,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":277,"thread_start_us":1189782,"threads_started":2}
I20260806 03:52:22.677093 6652 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:23.726433 6688 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:23.735539 2537 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 1.111s user 0.000s sys 0.014s
I20260806 03:52:22.711828 6687 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:23.749348 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 1.25676e+06us
W20260806 03:52:23.752009 3545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.277s user 0.000s sys 0.013s
W20260806 03:52:23.750058 2438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.616s user 0.000s sys 0.016s
W20260806 03:52:23.755067 2438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.631s user 0.000s sys 0.017s
I20260806 03:52:22.737327 6699 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:22.647096 6697 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:52:23.747067 6483 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.093s user 0.004s sys 0.027s
I20260806 03:52:22.562832 6700 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:23.686322 6533 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.775157 6028 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:23.828248 6587 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.769148 6648 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:23.830642 6648 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.831043 6603 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:23.831908 6603 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.831717 6495 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:23.833400 6648 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.834607 6622 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.836891 6673 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:23.837847 6622 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.839321 6447 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:23.839664 6650 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:23.840521 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 45400us
I20260806 03:52:23.843521 6650 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:23.796929 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 1.36982e+06us
I20260806 03:52:23.923964 6544 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:23.926111 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 150867us
I20260806 03:52:23.928689 2429 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:22.645545 6693 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 1.181s
I20260806 03:52:23.934459 6643 consensus_queue.cc:1035] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:23.936357 6700 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.963534 6603 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.961017 6697 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:23.970714 6674 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:23.915450 6699 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:23.971731 6361 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:23.960768 6587 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.954829 6533 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.977538 6682 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:23.977645 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 408) took 4556 ms. Trace:
I20260806 03:52:23.978144 6495 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:23.975831 6525 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.010s
I20260806 03:52:23.978410 2368 rpcz_store.cc:276] 0806 03:52:19.421510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:20.801016 (+1379506us) service_pool.cc:227] Handling call
0806 03:52:20.801471 (+ 455us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:21.619710 (+818239us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:21.619758 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:21.624506 (+ 4748us) log.cc:842] Serialized 26 byte log entry
0806 03:52:22.256836 (+632330us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:22.256885 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:22.256964 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:23.977092 (+1720128us) raft_consensus.cc:1671] finished
0806 03:52:23.977137 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:23.977398 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":158968,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":67705,"thread_start_us":1449376,"threads_started":2,"wal-append.queue_time_us":748551}
I20260806 03:52:23.988166 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 422) took 1561 ms. Trace:
I20260806 03:52:23.988705 2368 rpcz_store.cc:276] 0806 03:52:22.427192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:23.986624 (+1559432us) service_pool.cc:227] Handling call
0806 03:52:23.987299 (+ 675us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:23.987733 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:23.987767 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:23.987810 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:23.987857 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:23.987916 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:23.988072 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:23.990649 6657 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.995043 6697 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.979081 6538 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:23.987419 6699 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:23.998409 2372 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 9. Current term is 10. Ops: []
I20260806 03:52:23.999601 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 420) took 1573 ms. Trace:
W20260806 03:52:24.000466 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 218101us
I20260806 03:52:24.000349 2372 rpcz_store.cc:276] 0806 03:52:22.426579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:23.997323 (+1570744us) service_pool.cc:227] Handling call
0806 03:52:23.998035 (+ 712us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:23.999074 (+ 1039us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:23.999354 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:24.015996 2482 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.234s user 0.011s sys 0.004s
W20260806 03:52:24.016661 2482 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.337s user 0.012s sys 0.004s
I20260806 03:52:24.029318 6700 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:24.030290 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.658s user 0.008s sys 0.008s
W20260806 03:52:24.030831 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.915s user 0.008s sys 0.008s
W20260806 03:52:24.031772 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.323s user 0.010s sys 0.014s
W20260806 03:52:24.032529 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.472s user 0.011s sys 0.014s
W20260806 03:52:24.078504 2572 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.720s user 0.015s sys 0.006s
W20260806 03:52:24.121966 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 345955us
I20260806 03:52:24.122421 2572 raft_consensus.cc:2396] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260806 03:52:24.120257 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 430462us
W20260806 03:52:24.151635 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 223210us
I20260806 03:52:24.125500 6688 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:24.198724 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 187850us
I20260806 03:52:24.215021 6542 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:24.217620 6542 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.251446 6643 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:24.252349 6643 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:24.252458 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 382) took 1813 ms. Trace:
I20260806 03:52:24.252991 6643 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:24.218456 6533 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:24.255360 6533 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.256167 6740 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:24.196735 6538 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:24.258231 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 78121.7us
W20260806 03:52:24.311513 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 60152.7us
I20260806 03:52:24.257265 6740 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.287310 6682 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:24.257910 6709 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:52:24.304625 6526 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.823s user 0.033s sys 0.055s
I20260806 03:52:24.309144 2429 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:24.317065 6651 consensus_meta.cc:287] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.651s user 0.020s sys 0.000s
I20260806 03:52:24.324863 6164 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:24.327867 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 73775us
I20260806 03:52:24.253043 2482 rpcz_store.cc:276] 0806 03:52:22.438670 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.438812 (+ 142us) service_pool.cc:227] Handling call
0806 03:52:22.439228 (+ 416us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:22.439474 (+ 246us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 03:52:22.679730 (+240256us) log.cc:842] Serialized 14 byte log entry
0806 03:52:24.199633 (+1519903us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:24.218912 (+ 19279us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:24.251969 (+ 33057us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.252137 (+ 168us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:24.252307 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":146193,"prepare.queue_time_us":1229756,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":179251,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":17435,"thread_start_us":1576376,"threads_started":2}
W20260806 03:52:24.324010 6526 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.936s user 0.034s sys 0.055s
I20260806 03:52:24.320708 6648 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:24.338999 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 120263us
I20260806 03:52:24.346501 6734 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:24.314822 6688 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.327245 2534 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:24.341913 6028 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.010s
I20260806 03:52:24.352778 6709 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.353891 6310 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:24.349543 6734 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:24.361325 6734 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:24.355705 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:24.362082 6709 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.356400 6682 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.364096 3591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36484 (request call id 322) took 5396 ms. Trace:
I20260806 03:52:24.364584 6651 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:24.364897 6682 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.365279 6651 raft_consensus.cc:686] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:24.341172 6648 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.365108 6538 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:24.367759 6648 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.369571 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:24.369889 2478 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2268) took 5412 ms (client timeout 1565 ms). Trace:
W20260806 03:52:24.370455 2478 rpcz_store.cc:269] 0806 03:52:18.957545 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.991437 (+ 33892us) service_pool.cc:227] Handling call
0806 03:52:22.051218 (+3059781us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b58007c94b0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7ca40f63 00007f0d7caa4aba 00007f0d7cabc3af 00007f0d7cabdb4e 00007f0d7cac9228 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c
0806 03:52:24.369394 (+2318176us) spinlock_profiling.cc:243] Waited 1.82 s on lock 0x7b54002a08d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:24.369625 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6995535872}
I20260806 03:52:24.371512 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.347219 6533 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.374193 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.377533 2579 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260806 03:52:24.374411 2372 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:24.381392 2368 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 10: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:24.382516 6633 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:24.382928 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.383433 6633 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.379030 2579 pending_rounds.cc:76] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 3
I20260806 03:52:24.384691 2579 pending_rounds.cc:97] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 6.4
W20260806 03:52:24.386765 3547 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.024s user 0.021s sys 0.062s
I20260806 03:52:24.364588 3591 rpcz_store.cc:276] 0806 03:52:18.967840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.968052 (+ 212us) service_pool.cc:227] Handling call
0806 03:52:18.971211 (+ 3159us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:19.049305 (+ 78094us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.363331 (+5314026us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b5400580554. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:24.363728 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":154658688}
W20260806 03:52:24.387167 3547 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.105s user 0.022s sys 0.062s
I20260806 03:52:24.383800 2368 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:24.384277 6481 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:24.389128 6481 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.390914 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.391422 6633 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:24.397571 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 77576.5us
I20260806 03:52:24.375639 3545 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:24.391834 2369 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:24.385195 2579 raft_consensus.cc:2866] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:24.415138 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.407363 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 381) took 1971 ms. Trace:
W20260806 03:52:24.415100 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 63413.5us
I20260806 03:52:24.398200 6384 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:24.408347 6688 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.422600 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.391875 6481 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.417721 2477 rpcz_store.cc:276] 0806 03:52:22.437244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.437712 (+ 468us) service_pool.cc:227] Handling call
0806 03:52:22.438105 (+ 393us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:23.825790 (+1387685us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:23.825871 (+ 81us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:23.825973 (+ 102us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:23.826059 (+ 86us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.406834 (+580775us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54000afad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:24.406884 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:24.407123 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3018030976}
I20260806 03:52:24.413748 6542 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.366787 6651 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:24.432294 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 92462.8us
W20260806 03:52:24.540759 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 188338us
W20260806 03:52:24.544924 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 56689.8us
I20260806 03:52:24.475392 2369 raft_consensus.cc:1226] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:24.460472 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2240) took 2918 ms. Trace:
W20260806 03:52:24.529896 2581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 328) took 2387 ms (client timeout 1664 ms). Trace:
I20260806 03:52:24.472214 2368 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
W20260806 03:52:24.443184 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 55858us
W20260806 03:52:24.505172 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 71001.1us
I20260806 03:52:24.557648 2377 rpcz_store.cc:276] 0806 03:52:21.542027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:21.542183 (+ 156us) service_pool.cc:227] Handling call
0806 03:52:21.542680 (+ 497us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:22.047781 (+505101us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.442833 (+2395052us) spinlock_profiling.cc:243] Waited 95.9 ms on lock 0x7b5400012a54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:24.460246 (+ 17413us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":220514560}
I20260806 03:52:24.434703 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.424525 6384 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:24.547614 2534 leader_election.cc:398] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6.
I20260806 03:52:24.566334 2372 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
W20260806 03:52:24.517700 6726 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.142s user 0.002s sys 0.000s
I20260806 03:52:24.567554 2534 leader_election.cc:401] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260806 03:52:24.538856 6740 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:24.562825 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 147935us
W20260806 03:52:24.557972 2581 rpcz_store.cc:269] 0806 03:52:22.142544 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.142685 (+ 141us) service_pool.cc:227] Handling call
0806 03:52:24.529308 (+2386623us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b5400155c54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:24.529581 (+ 273us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":362261760}
I20260806 03:52:24.425355 6538 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:24.575174 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.449075 2480 raft_consensus.cc:2932] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 3 to 6, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:24.581545 2481 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8655b80017d7422082efde215e797b5b"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54220
I20260806 03:52:24.435721 6584 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:24.586632 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.584224 2377 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:24.593041 2481 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:52:24.567461 6384 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:24.686131 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.690311 2481 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:24.694446 6734 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:24.698809 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:24.691025 2332 connection.cc:232] RPC call timeout handler was delayed by -1.3800782118923962s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1652s for a response
I20260806 03:52:24.644114 2477 raft_consensus.cc:3032] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
W20260806 03:52:24.691833 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 85998.7us
W20260806 03:52:24.583317 6695 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 2.268s user 0.018s sys 0.030s
I20260806 03:52:24.712401 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2243) took 3086 ms. Trace:
I20260806 03:52:24.710459 2477 raft_consensus.cc:729] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:24.709692 2332 connection.cc:232] RPC call timeout handler was delayed by -1.3359782120678574s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17010000000000003s for a response
W20260806 03:52:24.715183 2332 connection.cc:232] RPC call timeout handler was delayed by -1.2378782120067626s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18100000000000002s for a response
W20260806 03:52:24.712352 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 165127us
W20260806 03:52:24.720571 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 119807us
I20260806 03:52:24.588593 6584 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:24.727960 2369 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:24.729158 6734 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:24.730430 2479 raft_consensus.cc:1226] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260806 03:52:24.731603 6212 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6.
I20260806 03:52:24.713095 2370 rpcz_store.cc:276] 0806 03:52:21.625871 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:21.626060 (+ 189us) service_pool.cc:227] Handling call
0806 03:52:21.819159 (+193099us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:23.387243 (+1568084us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.699821 (+1312578us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b54003322d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:24.712304 (+ 12483us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3603314560}
W20260806 03:52:24.738739 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0110782119445503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2062s for a response
W20260806 03:52:24.739670 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0101782120764256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20630000000000004s for a response
I20260806 03:52:24.744634 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.747082 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.747231 6740 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:24.748047 2482 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:24.748492 6740 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.749572 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.749294 6739 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260806 03:52:24.751293 2483 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2251) took 5772 ms (client timeout 1986 ms). Trace:
I20260806 03:52:24.751442 6740 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:24.751778 2483 rpcz_store.cc:269] 0806 03:52:18.951690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.953942 (+ 2252us) service_pool.cc:227] Handling call
0806 03:52:24.702961 (+5749019us) spinlock_profiling.cc:243] Waited 804 ms on lock 0x7b54001dc9d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:24.723901 (+ 20940us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1848339200}
I20260806 03:52:24.753157 6430 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:24.753616 6602 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:24.753885 6430 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:24.754382 6602 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.758212 6430 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:24.758857 6430 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:24.759361 6430 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:24.766460 6430 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:24.763881 6740 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:24.767625 6740 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.737246 6584 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:24.770349 6740 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.772859 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1104) took 2394 ms. Trace:
I20260806 03:52:24.773303 2579 rpcz_store.cc:276] 0806 03:52:22.378516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.379354 (+ 838us) service_pool.cc:227] Handling call
0806 03:52:22.379789 (+ 435us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:24.426430 (+2046641us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.772434 (+346004us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b540039bcd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:24.772762 (+ 328us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4588461440}
I20260806 03:52:24.760684 6542 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:24.761929 6643 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.508s user 0.015s sys 0.002s
I20260806 03:52:24.763163 6212 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:24.763360 2332 leader_election.cc:334] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.810s (SENT)
I20260806 03:52:24.643802 6636 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.790669 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.783128 2484 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:24.788005 2332 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:24.789469 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.784304 6734 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:24.767879 6376 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:24.805176 6615 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.806128 6615 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:24.806571 6536 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:24.799908 2477 consensus_queue.cc:260] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.819808 2478 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:52:24.825451 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.785998 6643 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.800930 6542 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.834304 3545 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:24.835543 2477 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:24.793725 6212 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.833572 6602 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.837981 6308 consensus_queue.cc:1046] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.029s
I20260806 03:52:24.839110 6308 raft_consensus.cc:3032] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:24.839483 6308 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:24.840607 6308 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:24.842016 6308 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:24.845131 2572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1103) took 2469 ms (client timeout 3205 ms). Trace:
I20260806 03:52:24.844467 6734 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:52:24.845614 2572 rpcz_store.cc:269] 0806 03:52:22.375680 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.376501 (+ 821us) service_pool.cc:227] Handling call
0806 03:52:24.844826 (+2468325us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b540024c2d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:24.845050 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2253263744}
W20260806 03:52:24.847347 2575 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1106) took 2423 ms (client timeout 2239 ms). Trace:
I20260806 03:52:24.850579 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 377) took 2427 ms. Trace:
I20260806 03:52:24.851027 2474 rpcz_store.cc:276] 0806 03:52:22.422998 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.427064 (+ 4066us) service_pool.cc:227] Handling call
0806 03:52:22.427482 (+ 418us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:23.834492 (+1407010us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:23.834547 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:23.834625 (+ 78us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:23.834691 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.850076 (+1015385us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54002b1454. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:24.850134 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:24.850325 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3234817408}
I20260806 03:52:24.854094 6542 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.858275 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 416) took 2439 ms. Trace:
I20260806 03:52:24.859047 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.859685 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.865087 2372 raft_consensus.cc:1261] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260806 03:52:24.858860 2379 rpcz_store.cc:276] 0806 03:52:22.424753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.428142 (+ 3389us) service_pool.cc:227] Handling call
0806 03:52:22.431419 (+ 3277us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:23.677436 (+1246017us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:23.677503 (+ 67us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:23.677578 (+ 75us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:23.677623 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.857754 (+1180131us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54000127d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:24.857806 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:24.858031 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2591795328}
I20260806 03:52:24.868405 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:24.847829 2575 rpcz_store.cc:269] 0806 03:52:22.424142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:22.443148 (+ 19006us) service_pool.cc:227] Handling call
0806 03:52:24.847070 (+2403922us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b5400238554. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:24.847259 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":307855616}
I20260806 03:52:24.874150 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.876091 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.874413 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.876274 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:24.874953 2482 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:24.865557 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:24.879279 2483 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 9: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:24.875864 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:24.884976 2484 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:24.886006 2579 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:24.887064 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.885499 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:24.891582 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 64307.4us
I20260806 03:52:24.891002 2370 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:24.899108 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.893754 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.896663 2478 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 9
I20260806 03:52:24.897519 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:24.903564 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:24.901114 2379 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:24.897895 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.905675 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.910516 6627 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:24.910579 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:24.911082 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:24.911134 6627 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:24.913157 6627 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:24.920642 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.925913 2369 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:24.898710 6734 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.923089 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 419) took 2496 ms. Trace:
I20260806 03:52:24.923367 2368 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:24.933169 2370 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:24.936205 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.931135 2374 rpcz_store.cc:276] 0806 03:52:22.426176 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:23.195997 (+769821us) service_pool.cc:227] Handling call
0806 03:52:23.272809 (+ 76812us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:23.731760 (+458951us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:23.731809 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:23.735478 (+ 3669us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:23.735541 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.922752 (+1187211us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b54004399d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:24.922820 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:24.922989 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":785202560}
I20260806 03:52:24.938793 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:24.941325 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.937549 2379 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:24.946285 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:24.948771 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.934742 2482 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:24.942346 2474 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:24.956615 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:24.959692 6205 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:24.961865 6643 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:24.962669 2583 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260806 03:52:24.966275 3594 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:24.966445 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:24.972990 2484 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:24.975271 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.976560 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:24.971598 2481 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:24.973688 3594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 449) took 5343 ms (client timeout 2138 ms). Trace:
W20260806 03:52:24.980439 3594 rpcz_store.cc:269] 0806 03:52:19.623756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.624094 (+ 338us) service_pool.cc:227] Handling call
0806 03:52:24.920605 (+5296511us) spinlock_profiling.cc:243] Waited 3.39 ms on lock 0x7b5800000f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:52:24.966769 (+ 46164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7787776}
I20260806 03:52:24.985199 2374 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
W20260806 03:52:24.987066 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 201840us
I20260806 03:52:24.986559 2369 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:24.990985 2481 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:25.005560 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.008914 2583 raft_consensus.cc:1226] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260806 03:52:25.015818 2477 raft_consensus.cc:1261] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260806 03:52:25.016396 2477 pending_rounds.cc:76] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 2
I20260806 03:52:25.016723 2477 pending_rounds.cc:97] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260806 03:52:25.017189 2477 raft_consensus.cc:2866] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:25.024503 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2213) took 6100 ms. Trace:
I20260806 03:52:25.025098 2375 rpcz_store.cc:276] 0806 03:52:18.829504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.829651 (+ 147us) service_pool.cc:227] Handling call
0806 03:52:18.830028 (+ 377us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:18.830436 (+ 408us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:18.830469 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:24.447564 (+5617095us) spinlock_profiling.cc:243] Waited 8.98 ms on lock 0x7b18002b83a0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b6af96 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a
0806 03:52:24.553530 (+105966us) spinlock_profiling.cc:243] Waited 33.7 ms on lock 0x7b2c00006f88. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:52:24.556433 (+ 2903us) log.cc:842] Serialized 26 byte log entry
0806 03:52:24.788125 (+231692us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:24.793307 (+ 5182us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:24.801429 (+ 8122us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:24.917227 (+115798us) raft_consensus.cc:1671] finished
0806 03:52:24.917253 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:24.929788 (+ 12535us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3030,"raft.run_cpu_time_us":208,"raft.run_wall_time_us":23577,"spinlock_wait_cycles":98223872,"thread_start_us":231033,"threads_started":1,"wal-append.queue_time_us":355390}
I20260806 03:52:25.026083 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.028398 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.036139 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:25.037108 2370 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:25.044111 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:25.046093 2481 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:25.045130 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:25.035324 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.056059 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.058390 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.062244 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.063054 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.044818 6553 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:25.064577 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:25.064862 6553 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.067107 2578 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:25.067157 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.070842 2482 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:25.077024 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.057612 6212 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:25.089257 2484 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:25.031817 6584 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.030876 6384 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.096061 2368 raft_consensus.cc:1261] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260806 03:52:25.091048 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.094003 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.093750 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:25.100056 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:25.100399 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54234 (request call id 2260) took 6138 ms. Trace:
I20260806 03:52:25.106057 2479 rpcz_store.cc:276] 0806 03:52:18.955844 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.030370 (+ 74526us) service_pool.cc:227] Handling call
0806 03:52:24.729841 (+5699471us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:24.730979 (+ 1138us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:25.093953 (+362974us) spinlock_profiling.cc:243] Waited 5.67 s on lock 0x7b54001dc9d0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:25.094159 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13047033088}
I20260806 03:52:25.117651 2479 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:25.132157 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.134801 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.135720 2574 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:25.138893 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.133404 2575 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:25.158878 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.162184 6698 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:25.162628 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.175072 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.185564 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.187004 2429 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:25.163247 6698 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.169097 6447 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:25.192205 6582 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:25.140307 6490 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:25.171676 6553 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:25.192095 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.206442 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.214509 2579 raft_consensus.cc:2396] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 7 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 2.
I20260806 03:52:25.214048 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.189741 6212 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:25.221880 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.207242 6688 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:25.234495 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:25.244429 2436 connection.cc:232] RPC call timeout handler was delayed by -3.9174733718391508s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21240000000000003s for a response
I20260806 03:52:25.246440 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.227198 6688 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:25.239745 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:52:25.205011 6490 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.228324 6447 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.254774 6677 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:25.258921 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.260882 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.261121 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.213517 6582 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.264812 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.260591 6447 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:25.233399 6212 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:25.251346 6688 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:25.253598 6584 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:25.255486 6677 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.271617 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.274092 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.268987 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.283847 3591 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 10.
I20260806 03:52:25.286516 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.280346 2429 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:25.287182 6490 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:25.287765 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.297243 2583 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:25.305281 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:25.287508 3588 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:25.314697 2538 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:25.313527 3590 raft_consensus.cc:2932] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Committing config change with OpId 7.6: config changed from index 3 to 6, cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:25.316833 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.318642 5510 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:25.319134 5510 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:25.319495 5510 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:25.327437 6212 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:25.302023 6384 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:25.329538 2538 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:25.336367 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 71324.1us
W20260806 03:52:25.338980 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 60520.1us
I20260806 03:52:25.341393 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.330235 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.344273 6643 raft_consensus.cc:2781] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:52:25.347059 2377 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
W20260806 03:52:25.347385 2436 connection.cc:232] RPC call timeout handler was delayed by -3.7906479919329286s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1634s for a response
I20260806 03:52:25.348004 2370 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:25.360090 2374 raft_consensus.cc:1226] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 1. Current term is 2. Ops: []
I20260806 03:52:25.361232 2538 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:25.360803 2478 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:25.371472 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.373073 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.373719 2584 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:25.374910 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.366566 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:25.379946 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:25.380999 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.383040 2429 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:25.395941 2572 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 6: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:25.397369 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.350100 6643 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:25.414719 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.416046 2574 pending_rounds.cc:76] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 6
I20260806 03:52:25.428769 2574 pending_rounds.cc:97] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 6.7
W20260806 03:52:25.420817 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 74640.6us
I20260806 03:52:25.415992 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.442006 6638 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:25.442281 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.442849 6638 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:25.443413 6638 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:25.445365 6627 consensus_meta.cc:287] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.532s user 0.012s sys 0.005s
I20260806 03:52:25.445889 6627 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.447250 3594 raft_consensus.cc:2379] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:25.384771 6384 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:25.429407 2574 raft_consensus.cc:2866] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260806 03:52:25.475642 2436 connection.cc:232] RPC call timeout handler was delayed by -2.8689733718056232s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3289s for a response
W20260806 03:52:25.643028 2436 connection.cc:232] RPC call timeout handler was delayed by -1.9476129058748484s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2239s for a response
I20260806 03:52:25.428786 6490 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:25.429102 6643 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:25.482198 2429 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:25.462118 2579 raft_consensus.cc:1226] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
W20260806 03:52:25.643587 2436 connection.cc:232] RPC call timeout handler was delayed by -1.0782129059080034s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3205s for a response
I20260806 03:52:25.561342 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.648103 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.472450 2577 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
W20260806 03:52:25.473855 6758 log.cc:924] Time spent T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.081s user 0.001s sys 0.001s
I20260806 03:52:25.666714 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.668481 6627 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:25.669265 6756 log.cc:924] Time spent T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.290s user 0.003s sys 0.000s
I20260806 03:52:25.565112 6698 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:25.536878 6531 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:25.671525 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.646517 6490 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:25.677910 2579 raft_consensus.cc:1226] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:25.459784 6384 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:25.679057 6535 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:52:25.679692 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 230766us
I20260806 03:52:25.689611 2534 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:25.738767 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 413) took 6200 ms. Trace:
I20260806 03:52:25.701253 2370 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
W20260806 03:52:25.800872 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 158387us
I20260806 03:52:25.826372 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.839561 2374 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
W20260806 03:52:25.845057 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 155606us
I20260806 03:52:25.855156 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:25.857164 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 55331.5us
I20260806 03:52:25.829404 6384 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:25.829591 6490 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:25.858563 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.859009 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.862789 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:25.858865 2429 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:25.851920 2538 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:25.828933 6535 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:25.823946 6531 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:25.865255 3585 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260806 03:52:25.865796 2370 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:25.860045 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:25.866285 2476 raft_consensus.cc:1261] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260806 03:52:25.866715 2476 pending_rounds.cc:76] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 6
I20260806 03:52:25.829873 2376 rpcz_store.cc:276] 0806 03:52:19.538182 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:19.538353 (+ 171us) service_pool.cc:227] Handling call
0806 03:52:19.538846 (+ 493us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:19.539341 (+ 495us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:19.539381 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:24.895972 (+5356591us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b18003ba7c0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b6af96 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a
0806 03:52:24.954444 (+ 58472us) log.cc:842] Serialized 26 byte log entry
0806 03:52:25.195560 (+241116us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:25.195613 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:25.195780 (+ 167us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:25.738153 (+542373us) raft_consensus.cc:1671] finished
0806 03:52:25.738183 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:25.738521 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3578,"raft.queue_time_us":137,"raft.run_cpu_time_us":210,"raft.run_wall_time_us":22696,"spinlock_wait_cycles":2667648,"thread_start_us":230273,"threads_started":1,"wal-append.queue_time_us":256541}
I20260806 03:52:25.866928 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:25.863457 2379 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:25.866981 2476 pending_rounds.cc:97] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260806 03:52:25.869649 2476 raft_consensus.cc:2866] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:25.879913 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:25.880203 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.881628 2377 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:25.756902 6688 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:25.885191 6688 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:25.885934 6688 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:25.886377 6688 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:25.886803 6688 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:25.891724 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.892112 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.862083 6698 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:25.905582 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:25.890415 5510 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.570s user 0.015s sys 0.003s
I20260806 03:52:25.895181 2379 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:25.914074 2578 raft_consensus.cc:1226] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:25.867046 6535 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:25.908697 2436 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:25.867215 6531 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:25.906826 2376 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:25.916996 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.912672 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.913735 5510 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:25.925153 2576 raft_consensus.cc:2379] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 6: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:25.896726 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:25.923120 2370 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:25.929774 3584 raft_consensus.cc:2354] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6.
I20260806 03:52:25.930728 5510 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:25.919024 6489 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:25.930845 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.931557 6489 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:25.932049 6489 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:25.933805 6643 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:25.938424 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.938573 6638 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:25.939841 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:25.941227 2578 raft_consensus.cc:1226] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:25.949848 2572 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:25.954818 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.958196 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.959028 2583 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:25.961090 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.962723 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.964563 2370 pending_rounds.cc:76] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 5
I20260806 03:52:25.965051 2370 pending_rounds.cc:97] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 7.6
I20260806 03:52:25.965386 2370 raft_consensus.cc:2866] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:25.966944 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.974148 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:25.978102 6489 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:25.979550 2579 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:25.981766 6489 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:25.984300 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 111771us
W20260806 03:52:25.968799 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 157076us
I20260806 03:52:25.989010 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.993126 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.993860 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.994604 2576 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 3.
I20260806 03:52:25.994846 2575 raft_consensus.cc:1226] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:25.997834 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:25.998322 2436 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:25.998777 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1207) took 1043 ms. Trace:
I20260806 03:52:25.999142 2373 rpcz_store.cc:276] 0806 03:52:24.955414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:25.393705 (+438291us) service_pool.cc:227] Handling call
0806 03:52:25.997568 (+603863us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65c6c 00007f0d79b65bd6 000000000034aea9 00007f0d81991a30 00007f0d81a577b4 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:52:25.998706 (+ 1138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":483349504}
I20260806 03:52:26.000800 6582 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:26.001771 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.990073 2578 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:26.005631 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.006505 2578 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:26.009827 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.008816 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:26.015115 2578 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:26.020594 2233 catalog_manager.cc:5495] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 2 to 4, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 4 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:26.027850 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:26.024338 2579 raft_consensus.cc:1226] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:26.018433 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.027064 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.028826 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:26.025650 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.035943 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:26.040629 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.040724 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:25.991552 6643 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:26.043574 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2262) took 1045 ms. Trace:
I20260806 03:52:26.068784 2369 rpcz_store.cc:276] 0806 03:52:24.998042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:25.402578 (+404536us) service_pool.cc:227] Handling call
0806 03:52:26.042544 (+639966us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:52:26.042837 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:26.043105 (+ 268us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:26.043144 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:26.043203 (+ 59us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:26.043245 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:26.043317 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:26.043486 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3225472}
W20260806 03:52:26.086956 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 46682us
I20260806 03:52:26.129096 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.130494 2581 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 9: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:26.133489 2233 catalog_manager.cc:5495] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 2 to 6, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 6 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:26.088824 6643 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:26.141350 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.152050 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.160136 2583 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:26.173291 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.181687 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.140713 6553 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:26.189939 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:26.161266 6643 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:26.200577 6384 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:26.201782 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:26.205569 6531 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.212819 2584 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:26.170782 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:26.219339 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:26.203359 6384 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:26.188035 6638 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:26.223984 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:26.183755 2233 catalog_manager.cc:5495] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 2 to 4, leader changed from cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 4 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:26.230635 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:26.208009 6531 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:26.242377 2233 catalog_manager.cc:5495] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 4 to 6, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 6 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:26.250988 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.250710 2233 catalog_manager.cc:5495] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 3 to 5, leader changed from cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 5 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:26.232507 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:52:26.255456 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 403056us
I20260806 03:52:26.221359 6553 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:26.257490 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 398676us
I20260806 03:52:26.223974 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:26.262522 2537 leader_election.cc:334] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.650s (SENT)
I20260806 03:52:26.262357 2233 catalog_manager.cc:5495] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
W20260806 03:52:26.263983 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 380804us
I20260806 03:52:26.262357 2429 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:26.266742 2537 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:26.275918 2574 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.007s sys 0.045s
I20260806 03:52:26.277225 2429 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:26.278644 2429 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:26.276620 2574 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s user 0.008s sys 0.045s
I20260806 03:52:26.281605 6643 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:26.279747 2233 catalog_manager.cc:5495] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:52:26.285832 2584 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260806 03:52:26.282178 6643 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:26.290588 2429 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:26.301236 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 449389us
W20260806 03:52:26.302392 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 439249us
I20260806 03:52:26.279439 6531 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:26.300438 2429 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:26.319530 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 57048.3us
I20260806 03:52:26.282387 6627 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:26.288925 6643 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:26.325416 2537 leader_election.cc:334] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.765s (SENT)
I20260806 03:52:26.280983 3585 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:26.284565 6384 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:26.336200 2538 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:26.331341 2537 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:26.322763 6531 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:26.342721 2534 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:26.346726 6531 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:26.347236 6775 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:26.347450 6638 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:26.347764 6775 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:26.326829 6627 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:26.348270 6638 raft_consensus.cc:686] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:26.348672 6627 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:26.349431 6638 consensus_queue.cc:237] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.328042 6677 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:26.351521 6677 consensus_queue.cc:1046] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260806 03:52:26.352551 6677 raft_consensus.cc:3032] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:26.353044 6677 raft_consensus.cc:729] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:26.348291 6775 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:26.353986 6553 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:26.354200 6677 consensus_queue.cc:260] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.355614 6677 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:26.346125 2538 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:26.358462 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 360271us
I20260806 03:52:26.362797 2534 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:26.363934 6373 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:26.364785 6013 consensus_queue.cc:1035] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260806 03:52:26.333592 6770 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:26.367034 6770 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:26.367446 6770 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:26.376525 6489 consensus_queue.cc:1035] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:26.379709 6613 consensus_queue.cc:1046] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260806 03:52:26.381029 6384 consensus_queue.cc:1046] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:52:26.382472 6772 raft_consensus.cc:3032] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:26.383013 6772 raft_consensus.cc:729] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:26.342018 6688 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.384481 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 337) took 1015 ms. Trace:
I20260806 03:52:26.385247 6688 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:26.385401 2331 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:26.386022 6688 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:26.386509 6688 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:26.386986 6688 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:26.384929 2574 rpcz_store.cc:276] 0806 03:52:25.369261 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:25.383983 (+ 14722us) service_pool.cc:227] Handling call
0806 03:52:25.415771 (+ 31788us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:25.461391 (+ 45620us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:25.461430 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:25.656446 (+195016us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b2c001765a8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:52:25.665633 (+ 9187us) log.cc:842] Serialized 26 byte log entry
0806 03:52:26.279813 (+614180us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:26.279856 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:26.279933 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:26.384147 (+104214us) raft_consensus.cc:1671] finished
0806 03:52:26.384171 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:26.384406 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2762112,"thread_start_us":599638,"threads_started":1,"wal-append.queue_time_us":702340}
I20260806 03:52:26.390391 6525 consensus_queue.cc:1046] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:26.389616 2233 catalog_manager.cc:5495] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 2 to 5, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 5 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:26.392728 2233 catalog_manager.cc:5495] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 4 to 5, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 5 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:52:26.344851 6535 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.398290 6772 consensus_queue.cc:260] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.397804 6698 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:26.398733 6536 raft_consensus.cc:3032] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:26.400003 6536 raft_consensus.cc:729] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:26.401077 6536 consensus_queue.cc:260] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.402841 6536 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:26.403623 6489 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:26.404533 6489 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:26.406127 6489 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:26.406422 6698 raft_consensus.cc:686] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:26.407675 6698 consensus_queue.cc:237] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.404647 6535 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:26.405591 6613 raft_consensus.cc:3032] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Stepping down as leader of term 1
I20260806 03:52:26.398317 2534 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:26.416347 2331 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:26.417899 2331 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:26.419598 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 189032us
W20260806 03:52:26.424598 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 99229.4us
W20260806 03:52:26.447599 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 150417us
I20260806 03:52:26.448606 2534 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:26.449744 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 77293.2us
I20260806 03:52:26.449157 6786 consensus_queue.cc:1046] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:52:26.450634 6553 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:26.462415 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1209) took 1046 ms. Trace:
I20260806 03:52:26.450603 6785 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:26.464140 3545 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:26.475073 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 245077us
I20260806 03:52:26.448951 2429 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:26.416119 6613 raft_consensus.cc:729] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:26.462960 2375 rpcz_store.cc:276] 0806 03:52:25.415730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:25.416616 (+ 886us) service_pool.cc:227] Handling call
0806 03:52:26.044356 (+627740us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:52:26.044629 (+ 273us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:26.088004 (+ 43375us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:26.088071 (+ 67us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:26.141666 (+ 53595us) log.cc:842] Serialized 26 byte log entry
0806 03:52:26.423246 (+281580us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:26.423329 (+ 83us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:26.423444 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:26.423515 (+ 71us) raft_consensus.cc:1671] finished
0806 03:52:26.423532 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:26.462334 (+ 38802us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7371,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":14803,"spinlock_wait_cycles":3280640,"thread_start_us":226315,"threads_started":1,"wal-append.queue_time_us":193762}
W20260806 03:52:26.484833 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 59311.3us
W20260806 03:52:26.486120 2537 leader_election.cc:334] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.897s (SENT)
I20260806 03:52:26.486655 2537 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:26.438882 6525 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:26.439239 6765 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:26.424476 6212 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:26.405784 6490 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:26.523729 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48829.9us
I20260806 03:52:26.473147 6553 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:26.473644 6785 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:26.433522 6535 consensus_queue.cc:1035] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260806 03:52:26.431914 6627 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:26.504966 6765 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:26.541049 2576 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:26.526479 6553 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:26.542024 2233 catalog_manager.cc:5495] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 1 to 2, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 2 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
W20260806 03:52:26.544973 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 68095.3us
I20260806 03:52:26.543701 6672 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:26.532632 6785 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:26.548063 6782 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 03:52:26.549809 6793 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:26.556226 6613 consensus_queue.cc:260] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.557826 6613 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:52:26.544104 6765 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:26.550472 6627 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:26.547873 6672 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:26.562297 2233 catalog_manager.cc:5495] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 1 to 5, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 5 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:26.552191 6490 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:26.547751 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 127243us
I20260806 03:52:26.574201 3545 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:26.596035 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 178220us
W20260806 03:52:26.605782 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 119719us
W20260806 03:52:26.606978 2537 leader_election.cc:334] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.978s (SENT)
W20260806 03:52:26.610088 2537 leader_election.cc:334] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.989s (SENT)
I20260806 03:52:26.612602 5828 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:26.613327 5828 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:26.614157 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.615443 6792 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:52:26.612845 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 115070us
I20260806 03:52:26.615650 5828 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:26.621230 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 60030.1us
I20260806 03:52:26.626066 2233 catalog_manager.cc:5495] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260806 03:52:26.622983 2429 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:26.633064 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:26.633912 2480 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:26.573720 6672 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:26.635798 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.640411 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:26.640488 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:26.648943 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 238016us
W20260806 03:52:26.650301 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 76297.2us
I20260806 03:52:26.651463 3545 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:26.654099 2476 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:26.652240 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:26.656217 2233 catalog_manager.cc:5495] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 2 to 3, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 3 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } }
I20260806 03:52:26.656231 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:26.659075 2477 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:26.661048 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:26.662025 2476 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:26.657162 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:26.668936 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:26.671798 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:26.674160 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 64133.1us
W20260806 03:52:26.675638 2537 leader_election.cc:334] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 2.059s (SENT)
I20260806 03:52:26.676496 2537 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:26.676625 2436 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:26.678066 2482 raft_consensus.cc:1226] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:26.677935 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.619850 6792 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:26.678783 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.679844 2483 raft_consensus.cc:2454] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:26.684715 2379 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:26.685602 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.681217 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:26.687191 2484 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:26.688885 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:26.683368 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.694202 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.694341 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:26.657893 6535 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:26.699910 2233 catalog_manager.cc:5495] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 reported cstate change: term changed from 1 to 5, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3). New cstate: current_term: 5 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } }
I20260806 03:52:26.804494 2481 raft_consensus.cc:1226] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:26.805019 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.684000 6582 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:52:26.643427 6775 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.626130 6627 consensus_queue.cc:1035] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260806 03:52:26.812992 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:26.690310 6792 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:26.841462 3590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 453) took 1559 ms. Trace:
I20260806 03:52:26.841813 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:26.842052 3590 rpcz_store.cc:276] 0806 03:52:25.270139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:25.307080 (+ 36941us) service_pool.cc:227] Handling call
0806 03:52:25.312697 (+ 5617us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:25.313117 (+ 420us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:52:25.657747 (+344630us) log.cc:842] Serialized 14 byte log entry
0806 03:52:26.491559 (+833812us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:26.491629 (+ 70us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:26.491698 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:26.491799 (+ 101us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:26.828873 (+337074us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b5400732650. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:26.828997 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15442,"raft.queue_time_us":197490,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":140,"spinlock_wait_cycles":11607936,"thread_start_us":623875,"threads_started":2,"wal-append.queue_time_us":479380}
I20260806 03:52:26.817719 6783 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:26.819445 6525 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:26.844954 6525 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:26.833858 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:26.704336 6535 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:26.802989 6786 raft_consensus.cc:3032] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
W20260806 03:52:26.836282 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 159721us
W20260806 03:52:26.855445 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 204095us
I20260806 03:52:26.856649 3545 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:26.860194 2480 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:26.863210 2482 raft_consensus.cc:1226] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: []
I20260806 03:52:26.693084 6643 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:26.870446 2436 leader_election.cc:398] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 8. Current term is 9.
I20260806 03:52:26.873536 2436 leader_election.cc:401] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
W20260806 03:52:26.871129 2332 leader_election.cc:334] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.565s (SENT)
W20260806 03:52:26.879040 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 203510us
I20260806 03:52:26.878512 2477 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:26.874264 2332 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:26.883670 2537 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:26.853528 6535 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:26.886675 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:26.887651 2482 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:26.854110 6786 raft_consensus.cc:729] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:26.881114 6783 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:26.898993 2484 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:26.935734 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:26.904153 6373 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:26.939563 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:26.904564 6783 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:26.951646 6801 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:26.952567 6789 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:26.939582 6799 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:26.888695 6803 consensus_queue.cc:1046] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
W20260806 03:52:26.966239 6677 consensus_meta.cc:287] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.610s user 0.016s sys 0.000s
I20260806 03:52:26.968060 6809 consensus_queue.cc:1046] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
W20260806 03:52:26.968792 6531 consensus_meta.cc:287] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.622s user 0.013s sys 0.000s
I20260806 03:52:26.968604 6808 consensus_queue.cc:1046] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:52:26.969470 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:26.971215 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:26.971474 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:26.980453 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:26.981415 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:26.988834 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 62860.1us
I20260806 03:52:27.039410 3586 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:27.012804 6799 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:26.990121 6783 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:26.935320 6794 consensus_queue.cc:1046] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.098s
W20260806 03:52:27.041208 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 191656us
W20260806 03:52:27.042280 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 383309us
I20260806 03:52:27.016299 2476 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:27.046211 2379 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:27.057411 2370 raft_consensus.cc:1226] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: []
W20260806 03:52:27.040578 6770 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.673s user 0.014s sys 0.000s
W20260806 03:52:27.071746 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 448908us
I20260806 03:52:27.072604 2429 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:27.076686 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 453727us
W20260806 03:52:27.063218 6489 consensus_meta.cc:287] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.656s user 0.011s sys 0.003s
I20260806 03:52:27.045545 6799 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:27.026880 6531 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.078009 3584 raft_consensus.cc:2379] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 9: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:27.003048 6786 consensus_queue.cc:260] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.077126 6775 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:27.110986 2482 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:27.125260 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:27.028128 6801 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:27.140727 2474 raft_consensus.cc:1226] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
W20260806 03:52:27.126448 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 58460.9us
I20260806 03:52:27.082290 6643 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:27.141877 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:27.146020 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 274909us
I20260806 03:52:27.149654 3594 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260806 03:52:27.146157 3590 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 10.
W20260806 03:52:27.151863 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 111840us
I20260806 03:52:27.157938 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:27.165009 2538 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:27.162322 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 94601.8us
I20260806 03:52:27.175136 2331 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.173413 2369 raft_consensus.cc:1261] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:52:27.107493 6489 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:27.142899 6787 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 8. Current term is 9.
I20260806 03:52:27.082842 6770 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:27.116218 5510 consensus_queue.cc:1046] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260806 03:52:27.188320 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:27.157943 6786 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:27.190191 2436 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:27.194587 2576 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:27.119230 6802 consensus_queue.cc:1035] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
I20260806 03:52:27.208772 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
W20260806 03:52:27.181843 6536 consensus_meta.cc:287] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.778s user 0.012s sys 0.004s
I20260806 03:52:27.209779 3594 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:27.213579 6531 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:27.220103 6800 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:27.220907 6800 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.222221 6819 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:27.222764 6819 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:27.223191 6819 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:27.229382 6525 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.182709 6803 raft_consensus.cc:3032] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:27.184638 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:27.231464 6817 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 03:52:27.233747 6772 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:27.236042 6809 raft_consensus.cc:3032] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:27.236536 6809 raft_consensus.cc:729] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:27.237586 6809 consensus_queue.cc:260] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:27.239403 6809 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:27.231475 6803 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:27.245863 6803 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.247365 6803 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:27.213167 6818 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
I20260806 03:52:27.251652 6836 consensus_queue.cc:1046] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260806 03:52:27.205668 6688 consensus_meta.cc:287] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.818s user 0.011s sys 0.004s
I20260806 03:52:27.252978 6836 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:27.253388 6688 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.253997 6836 raft_consensus.cc:3032] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:27.254365 6836 raft_consensus.cc:729] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:27.188918 6810 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.242s
I20260806 03:52:27.229764 6770 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:27.261363 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:27.277469 6770 raft_consensus.cc:2741] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 03:52:27.277668 6829 consensus_queue.cc:1046] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260806 03:52:27.283535 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 103530us
I20260806 03:52:27.267967 3586 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
W20260806 03:52:27.288359 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 215832us
I20260806 03:52:27.289382 2429 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:27.195478 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:27.302711 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 127665us
I20260806 03:52:27.305325 2331 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:27.270083 6613 consensus_meta.cc:287] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.712s user 0.011s sys 0.004s
W20260806 03:52:27.311152 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 121098us
I20260806 03:52:27.313910 2436 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.316577 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:27.317763 6525 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:27.318259 6841 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:27.257474 6489 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:27.318512 6525 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:52:27.318914 6841 raft_consensus.cc:686] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:27.319980 6841 consensus_queue.cc:237] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:27.322250 6553 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.753s user 0.014s sys 0.000s
I20260806 03:52:27.322821 6553 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.256016 6688 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:27.255260 6836 consensus_queue.cc:260] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.324430 2429 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:27.328863 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:27.360984 2374 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:27.324080 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 139848us
I20260806 03:52:27.285329 6373 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:27.304206 2332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.513s (SENT) [suppressed 58 similar messages]
W20260806 03:52:27.376250 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 61263.5us
W20260806 03:52:27.385272 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 71465.2us
I20260806 03:52:27.388425 2331 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:27.306844 6770 raft_consensus.cc:2741] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260806 03:52:27.398885 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:27.399856 3594 raft_consensus.cc:1261] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260806 03:52:27.366041 6785 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.806s user 0.011s sys 0.005s
W20260806 03:52:27.408454 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 224995us
I20260806 03:52:27.309636 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:27.404959 6837 consensus_queue.cc:1046] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260806 03:52:27.426887 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 56713.9us
W20260806 03:52:27.427122 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 51970.3us
W20260806 03:52:27.402292 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 98129.2us
I20260806 03:52:27.416968 2436 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.434427 2429 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.383894 6801 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:27.399770 6672 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.730s user 0.013s sys 0.000s
I20260806 03:52:27.437635 2436 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:27.361353 6490 consensus_queue.cc:1046] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260806 03:52:27.421059 6818 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:27.472240 6818 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:27.472816 6818 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:27.468439 6801 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:52:27.473495 6801 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:27.473958 6801 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:52:27.472895 6531 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:27.420168 6810 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260806 03:52:27.438225 6792 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.596s user 0.016s sys 0.000s
I20260806 03:52:27.433733 6836 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:27.475945 2374 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
W20260806 03:52:27.485786 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 51468.6us
I20260806 03:52:27.488327 2538 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:27.452404 6672 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:27.507025 6815 raft_consensus.cc:2781] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:27.517930 6536 consensus_queue.cc:1046] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:52:27.518535 6553 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:27.503779 2332 leader_election.cc:334] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.652s (SENT)
I20260806 03:52:27.525893 2332 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:27.491451 2436 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.498994 2429 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.466014 6817 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:27.489796 6688 raft_consensus.cc:2741] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 03:52:27.534513 2574 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:27.430406 6785 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
W20260806 03:52:27.505429 6765 consensus_meta.cc:287] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.945s user 0.010s sys 0.006s
W20260806 03:52:27.559651 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 60740us
I20260806 03:52:27.536504 3591 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:27.520475 6582 raft_consensus.cc:3032] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:27.560871 2429 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:27.565871 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 62142.3us
W20260806 03:52:27.574508 2332 leader_election.cc:334] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.701s (SENT)
I20260806 03:52:27.575413 2332 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:27.577718 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 89531.4us
I20260806 03:52:27.578851 2538 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:27.586236 2538 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:27.588066 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:27.589248 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.592558 2375 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 9: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:27.596282 2476 raft_consensus.cc:1226] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:27.549696 6810 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:27.600865 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.517990 6815 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:27.599501 2483 raft_consensus.cc:1226] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:27.596004 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.601600 2482 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:27.674669 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.601567 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.603327 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:27.604579 6789 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:27.631927 6815 raft_consensus.cc:3037] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:27.639902 6645 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:27.596724 6829 consensus_queue.cc:1035] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260806 03:52:27.568812 6582 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:27.558892 6819 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:27.689755 6799 consensus_meta.cc:287] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.573s user 0.011s sys 0.002s
I20260806 03:52:27.640076 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.590840 6384 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:27.592166 6490 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:27.600481 6677 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:27.570228 6531 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:27.677286 2477 raft_consensus.cc:2454] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:27.720624 6525 raft_consensus.cc:3032] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:27.721099 6525 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:27.721582 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.722414 2477 raft_consensus.cc:2379] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:27.722119 6525 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:27.723248 6786 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.518s user 0.007s sys 0.002s
I20260806 03:52:27.598407 6688 raft_consensus.cc:2781] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:52:27.681110 2481 raft_consensus.cc:1226] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:27.713708 2534 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:27.682782 2476 raft_consensus.cc:1226] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:27.686926 2474 raft_consensus.cc:1226] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
W20260806 03:52:27.671814 6535 consensus_meta.cc:287] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.760s user 0.014s sys 0.002s
W20260806 03:52:27.695322 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 120826us
I20260806 03:52:27.736909 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.631377 6373 consensus_queue.cc:1046] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260806 03:52:27.706024 6789 raft_consensus.cc:686] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:27.723958 6627 raft_consensus.cc:3032] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:27.737774 2478 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
W20260806 03:52:27.583274 6823 log.cc:924] Time spent T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260806 03:52:27.493909 6840 consensus_queue.cc:1046] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.107s
I20260806 03:52:27.578920 6765 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.640630 2575 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:27.790441 2575 raft_consensus.cc:595] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:27.790874 2575 raft_consensus.cc:603] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:27.754382 6490 raft_consensus.cc:686] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:27.791479 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:27.705032 2576 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:27.715260 6672 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:27.715706 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 549288us
I20260806 03:52:27.760851 6627 raft_consensus.cc:729] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:27.796097 2537 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:27.717990 6793 raft_consensus.cc:3032] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
W20260806 03:52:27.718623 6783 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.656s user 0.004s sys 0.009s
I20260806 03:52:27.719269 6796 raft_consensus.cc:3032] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:27.719221 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:27.709623 6645 raft_consensus.cc:686] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:27.804688 6772 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.570s user 0.012s sys 0.003s
I20260806 03:52:27.804863 6645 consensus_queue.cc:237] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.805567 6772 raft_consensus.cc:3032] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:27.805958 6772 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:27.806218 6840 raft_consensus.cc:3032] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Stepping down as leader of term 2
I20260806 03:52:27.806674 6840 raft_consensus.cc:729] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:27.808130 6843 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:27.803747 3587 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:27.809832 6490 consensus_queue.cc:237] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.811748 6796 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:27.813215 6841 raft_consensus.cc:3032] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:27.812922 6796 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.813650 6841 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:27.813745 6789 consensus_queue.cc:237] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.814487 6796 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:27.814666 6841 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.816421 6841 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:27.820492 6783 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:27.821429 6765 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:27.823295 6783 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:27.824170 3586 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:27.807475 6772 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.824437 6013 raft_consensus.cc:3032] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:27.824834 6013 raft_consensus.cc:729] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:27.758363 6582 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:27.825721 6013 consensus_queue.cc:260] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.826550 6582 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:27.798668 6793 raft_consensus.cc:729] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:27.755177 2476 raft_consensus.cc:1226] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:27.828172 6793 consensus_queue.cc:260] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.828775 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.829553 6793 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:27.830121 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.831432 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.755676 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:27.706162 6855 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:27.767343 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:27.832899 2374 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:27.833841 2537 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:27.709295 6799 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.785914 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:27.843338 2376 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:27.785820 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:27.786875 2332 leader_election.cc:334] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.062s (SENT)
I20260806 03:52:27.844892 2478 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:27.758512 6212 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:27.750571 6688 raft_consensus.cc:686] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:27.789462 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:27.793150 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 63315.2us
W20260806 03:52:27.794092 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 366116us
I20260806 03:52:27.861371 2331 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.776705 6785 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:27.779126 6384 raft_consensus.cc:686] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:27.920390 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 59120.4us
I20260806 03:52:27.753026 6535 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.697850 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:27.928757 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:27.929251 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 231485us
I20260806 03:52:27.930799 3547 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.827122 6013 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:27.489889 6792 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:27.844722 2332 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.785816 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:27.737463 6780 consensus_queue.cc:1046] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.094s
I20260806 03:52:27.851125 2374 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:27.923486 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.936967 2480 raft_consensus.cc:1226] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:27.938328 2484 raft_consensus.cc:1226] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
W20260806 03:52:27.937243 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 103542us
W20260806 03:52:27.848995 6803 consensus_meta.cc:287] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.594s user 0.007s sys 0.007s
I20260806 03:52:27.940128 2537 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:27.846835 6797 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:27.956759 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:27.942382 2478 raft_consensus.cc:2379] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:27.960813 2331 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:27.860194 5828 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:52:27.963373 2483 raft_consensus.cc:1226] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:27.963171 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.921638 6820 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.038738 6819 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:28.210526 6818 consensus_meta.cc:287] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.711s user 0.014s sys 0.003s
I20260806 03:52:28.048810 2368 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:28.011059 6854 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 03:52:28.011015 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 229372us
W20260806 03:52:28.012245 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 47179.8us
W20260806 03:52:28.302685 2332 leader_election.cc:334] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.063s (SENT)
I20260806 03:52:28.303243 2332 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:28.303373 2537 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:27.958843 6765 consensus_queue.cc:1046] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.114s
I20260806 03:52:28.313213 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.021728 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.940887 6855 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:27.807595 6840 consensus_queue.cc:260] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.019340 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:27.956099 6672 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:52:28.088189 6862 consensus_queue.cc:1046] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260806 03:52:28.087132 6845 consensus_queue.cc:1046] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.053s
I20260806 03:52:28.031153 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:28.098157 6535 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:28.192540 6831 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:28.096411 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2213) took 1489 ms. Trace:
I20260806 03:52:27.958338 6783 consensus_queue.cc:1046] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260806 03:52:27.936698 6525 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:27.921973 6212 raft_consensus.cc:686] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:28.285276 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 324585us
I20260806 03:52:28.029698 6802 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:28.131318 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1216) took 1477 ms. Trace:
I20260806 03:52:28.189002 6825 raft_consensus.cc:3032] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:28.047163 6861 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:28.036065 6627 consensus_queue.cc:260] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:28.287003 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2226) took 1353 ms. Trace:
W20260806 03:52:28.283888 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 386054us
I20260806 03:52:28.181473 6803 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:28.293583 2476 raft_consensus.cc:2379] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:28.297652 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 366936us
W20260806 03:52:28.283375 6553 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.001s sys 0.010s
I20260806 03:52:28.308043 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:28.311425 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1217) took 1661 ms. Trace:
I20260806 03:52:28.129639 6688 consensus_queue.cc:237] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.314203 2483 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:28.314407 2575 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:28.354482 2368 rpcz_store.cc:276] 0806 03:52:26.649731 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:26.650031 (+ 300us) service_pool.cc:227] Handling call
0806 03:52:26.679432 (+ 29401us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:28.299809 (+1620377us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:28.311217 (+ 11408us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b5400012a54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:28.311371 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2965346048}
I20260806 03:52:28.317381 2537 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:28.286787 6841 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.326352 2478 raft_consensus.cc:2379] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.359175 2429 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:28.318691 2474 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:28.324884 2479 rpcz_store.cc:276] 0806 03:52:26.607322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:26.607449 (+ 127us) service_pool.cc:227] Handling call
0806 03:52:26.616892 (+ 9443us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:26.617159 (+ 267us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:26.617184 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:26.617976 (+ 792us) log.cc:842] Serialized 26 byte log entry
0806 03:52:27.081694 (+463718us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:27.081744 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:27.083381 (+ 1637us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:28.096108 (+1012727us) raft_consensus.cc:1671] finished
0806 03:52:28.096137 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:28.096312 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":460912,"threads_started":1,"wal-append.queue_time_us":754210}
I20260806 03:52:28.323932 6373 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:28.331758 6832 consensus_queue.cc:1046] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260806 03:52:28.299065 6820 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.303470 6818 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.299173 6816 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.323884 6843 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:28.157029 6384 consensus_queue.cc:237] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:28.337126 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.302474 6772 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:28.315631 6809 consensus_meta.cc:287] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.076s user 0.012s sys 0.008s
I20260806 03:52:28.348281 6799 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:28.344339 2375 rpcz_store.cc:276] 0806 03:52:26.649243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:26.649424 (+ 181us) service_pool.cc:227] Handling call
0806 03:52:28.126532 (+1477108us) spinlock_profiling.cc:243] Waited 923 ms on lock 0x7b5400012cd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:28.126796 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2121769856}
I20260806 03:52:28.336231 6792 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:28.315480 6800 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:28.389391 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.349279 2484 rpcz_store.cc:276] 0806 03:52:26.933909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:26.934076 (+ 167us) service_pool.cc:227] Handling call
0806 03:52:26.934388 (+ 312us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:27.948684 (+1014296us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:28.127903 (+179219us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54002701d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:28.286896 (+158993us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2308420352}
I20260806 03:52:28.322157 6672 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:28.347451 2332 leader_election.cc:334] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.090s (SENT)
I20260806 03:52:28.355175 2575 raft_consensus.cc:595] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:28.286526 6770 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.358137 2379 raft_consensus.cc:2379] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:28.362246 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:28.362388 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.363612 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:28.368072 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.369537 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:28.351142 6785 consensus_queue.cc:1035] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260806 03:52:28.348905 6825 raft_consensus.cc:729] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:28.366802 6841 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:28.372184 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 54922.6us
I20260806 03:52:28.431097 2537 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:28.320953 6698 consensus_queue.cc:1046] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260806 03:52:28.306493 6854 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
W20260806 03:52:28.354045 6553 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.618s user 0.002s sys 0.010s
W20260806 03:52:28.364651 6836 consensus_meta.cc:287] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.877s user 0.014s sys 0.000s
I20260806 03:52:28.380599 6765 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:28.362519 6373 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:28.373754 6627 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:28.391721 2376 consensus_meta.cc:287] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.531s user 0.006s sys 0.006s
I20260806 03:52:28.444192 2376 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:28.381032 2480 raft_consensus.cc:2454] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:28.409471 2332 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:28.416177 2483 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 1 FOLLOWER]: Advancing to term 2
I20260806 03:52:28.415141 2476 raft_consensus.cc:2362] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8. Re-sending same reply.
I20260806 03:52:28.419344 2478 raft_consensus.cc:2379] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.371973 6535 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:28.340975 6831 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.433233 2370 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:28.433693 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.368602 6851 consensus_queue.cc:1046] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260806 03:52:28.438122 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.439196 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.415114 6809 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.446642 2368 raft_consensus.cc:1226] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:28.448050 3545 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:28.366818 6802 consensus_queue.cc:1046] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260806 03:52:28.455834 2479 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:28.381034 6816 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.418208 6672 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Advancing to term 11
W20260806 03:52:28.469378 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 121962us
I20260806 03:52:28.435977 6854 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:52:28.480481 2332 leader_election.cc:334] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.092s (SENT)
I20260806 03:52:28.480983 2332 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:28.482565 6836 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:28.470038 2369 raft_consensus.cc:1226] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:28.464758 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:28.484367 2478 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:28.472766 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.478547 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.488682 2480 raft_consensus.cc:2379] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.489689 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.490613 2480 raft_consensus.cc:2396] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 03:52:28.491439 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.443475 6373 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:28.488682 2476 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.501240 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:28.497992 2478 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:28.503892 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.500514 2480 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 9: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:28.501842 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.513007 2484 raft_consensus.cc:2379] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.513712 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.424532 6770 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.514580 2478 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:28.522737 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.523833 2474 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:28.388413 6212 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:28.531744 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.529043 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.534839 2479 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.469018 6535 raft_consensus.cc:686] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:28.469625 6792 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:28.478034 6783 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:28.530526 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.532894 2478 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:28.549075 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.550580 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.500551 6797 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.510663 6850 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.562718 2480 raft_consensus.cc:2379] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.563862 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:28.544780 2476 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
I20260806 03:52:28.505541 6553 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:28.576426 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.606237 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.613308 2476 raft_consensus.cc:2379] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.606519 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:28.501361 6013 consensus_queue.cc:1046] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.061s
I20260806 03:52:28.613263 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.623683 2479 raft_consensus.cc:2379] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.627060 2480 raft_consensus.cc:1226] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:28.627764 2484 raft_consensus.cc:1226] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
W20260806 03:52:28.632440 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 149818us
W20260806 03:52:28.632799 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 73206.8us
I20260806 03:52:28.633659 2534 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:28.633154 2478 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:28.474392 6809 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:28.639748 2375 raft_consensus.cc:1226] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:28.640259 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.647040 2484 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:28.569075 6783 raft_consensus.cc:686] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:28.661768 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:28.666059 2429 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:28.667200 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:28.667629 2332 leader_election.cc:334] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.579s (SENT)
I20260806 03:52:28.688525 2474 raft_consensus.cc:1261] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 03:52:28.691459 2332 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:28.695096 2479 raft_consensus.cc:1226] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6. Ops: []
W20260806 03:52:28.663406 6582 consensus_meta.cc:287] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.836s user 0.008s sys 0.008s
I20260806 03:52:28.700878 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:28.651002 6801 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.177s user 0.011s sys 0.008s
I20260806 03:52:28.632268 6845 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:28.719820 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:28.702713 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.723376 2480 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
W20260806 03:52:28.726361 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58792.1us
I20260806 03:52:28.726733 2375 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 9: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:52:28.727682 2332 leader_election.cc:334] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.721s (SENT)
I20260806 03:52:28.728300 2332 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:28.669217 6785 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.535661 6825 consensus_queue.cc:260] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:28.681824 6818 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:28.738140 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:28.678436 6843 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.578729 6850 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.740962 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.759581 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:28.760123 2480 raft_consensus.cc:2379] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:28.741175 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 135910us
I20260806 03:52:28.749063 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:28.700538 6800 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:28.570266 6797 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:28.703223 6862 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260806 03:52:28.781210 6870 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.352s user 0.001s sys 0.001s
I20260806 03:52:28.728811 6803 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:28.786049 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.787154 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.862845 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.719815 6820 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:28.795600 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 67867us
W20260806 03:52:28.889740 2332 leader_election.cc:334] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.724s (SENT)
I20260806 03:52:28.805330 2479 raft_consensus.cc:1226] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:28.860357 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.863915 2478 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:28.897179 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.899092 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.678177 6688 consensus_queue.cc:1046] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
W20260806 03:52:28.911451 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 276537us
I20260806 03:52:28.900671 2478 raft_consensus.cc:2379] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.913975 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.914878 2478 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:28.747761 6841 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:28.921123 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:28.901937 2332 leader_election.cc:334] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.516s (SENT)
W20260806 03:52:28.940006 2332 leader_election.cc:334] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.538s (SENT)
I20260806 03:52:28.940608 2332 leader_election.cc:302] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:28.903496 2484 raft_consensus.cc:2379] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.697052 6836 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.091s
W20260806 03:52:28.942829 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 237315us
I20260806 03:52:28.800853 6825 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:28.761221 6698 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:28.697000 6861 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.216s
I20260806 03:52:29.012761 2534 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:29.013327 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.729106 6801 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:28.944422 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:29.024595 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.021490 6489 raft_consensus.cc:3032] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Stepping down as leader of term 3
W20260806 03:52:29.061194 6885 log.cc:924] Time spent T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.367s user 0.000s sys 0.001s
W20260806 03:52:29.062098 6877 log.cc:924] Time spent T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.438s user 0.001s sys 0.000s
W20260806 03:52:29.063099 6878 log.cc:924] Time spent T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.272s user 0.000s sys 0.000s
W20260806 03:52:29.063776 6865 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.550s user 0.000s sys 0.001s
I20260806 03:52:28.912791 6818 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.021349 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:29.044827 6698 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:28.745057 6535 consensus_queue.cc:237] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.083148 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:28.735771 6845 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.126699 6816 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:29.057917 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 117946us
I20260806 03:52:29.091625 2484 raft_consensus.cc:2379] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:28.798532 6792 raft_consensus.cc:2726] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:29.095326 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:29.098624 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 85992.8us
W20260806 03:52:28.913682 6772 consensus_meta.cc:287] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.531s user 0.013s sys 0.000s
W20260806 03:52:29.101953 2483 consensus_meta.cc:287] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.633s user 0.009s sys 0.005s
I20260806 03:52:29.254061 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.235771 6818 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:28.932394 6868 log.cc:924] Time spent T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.493s user 0.000s sys 0.002s
I20260806 03:52:29.192512 2377 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 11: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:29.085561 6841 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:28.788161 6843 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.223773 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 404) took 1586 ms. Trace:
I20260806 03:52:28.938638 6803 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.095155 6770 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:28.748812 6785 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:29.236783 2482 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2277) took 1575 ms (client timeout 2063 ms). Trace:
I20260806 03:52:29.094080 6489 raft_consensus.cc:729] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:29.089174 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 299682us
I20260806 03:52:28.932228 6799 consensus_queue.cc:1046] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.300s
I20260806 03:52:29.149902 5510 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:29.258468 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:29.260661 2537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.653s user 0.002s sys 0.011s
W20260806 03:52:29.374644 2537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.767s user 0.003s sys 0.011s
I20260806 03:52:29.255492 2483 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 2.
W20260806 03:52:29.376313 2483 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2294) took 1286 ms (client timeout 1662 ms). Trace:
I20260806 03:52:29.343433 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:29.377144 2483 rpcz_store.cc:269] 0806 03:52:28.090418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.366313 (+275895us) service_pool.cc:227] Handling call
0806 03:52:29.376207 (+1009894us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:29.377521 2478 raft_consensus.cc:2362] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7. Re-sending same reply.
I20260806 03:52:29.262691 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.193392 6831 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.385798 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.277530 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:54212 (request call id 458) took 1574 ms. Trace:
I20260806 03:52:29.275316 2481 rpcz_store.cc:276] 0806 03:52:27.637878 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:27.732514 (+ 94636us) service_pool.cc:227] Handling call
0806 03:52:27.754384 (+ 21870us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:27.778921 (+ 24537us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:27.778960 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:27.798055 (+ 19095us) log.cc:842] Serialized 487 byte log entry
0806 03:52:28.170089 (+372034us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:28.170136 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:28.170224 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:29.223418 (+1053194us) raft_consensus.cc:1671] finished
0806 03:52:29.223455 (+ 37us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:29.223675 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":250911,"threads_started":1,"wal-append.queue_time_us":562532}
I20260806 03:52:29.252532 6836 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:29.285710 2482 rpcz_store.cc:269] 0806 03:52:27.650033 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:27.674320 (+ 24287us) service_pool.cc:227] Handling call
0806 03:52:29.224945 (+1550625us) spinlock_profiling.cc:243] Waited 764 ms on lock 0x7b5400150c54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:29.225167 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1756297984}
W20260806 03:52:29.370968 2538 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.841s user 0.000s sys 0.019s
W20260806 03:52:29.371743 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 301154us
W20260806 03:52:29.380539 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 112980us
I20260806 03:52:29.375097 2484 raft_consensus.cc:2379] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:29.391577 2476 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.632s user 0.054s sys 0.041s
I20260806 03:52:29.055133 6780 raft_consensus.cc:3032] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:28.935031 6783 consensus_queue.cc:237] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.237211 6902 raft_consensus.cc:2726] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:29.388211 2474 rpcz_store.cc:276] 0806 03:52:27.695554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.757769 (+1062215us) service_pool.cc:227] Handling call
0806 03:52:29.263961 (+506192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:29.387564 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:29.391644 2538 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.861s user 0.000s sys 0.020s
W20260806 03:52:29.382649 2332 leader_election.cc:334] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.513s (SENT)
I20260806 03:52:29.276854 6857 raft_consensus.cc:3032] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:29.270161 6899 raft_consensus.cc:3032] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Stepping down as leader of term 8
W20260806 03:52:29.392940 2476 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s user 0.054s sys 0.041s
W20260806 03:52:29.255988 6802 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.724s user 0.006s sys 0.007s
I20260806 03:52:29.345172 6772 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.418759 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:29.378945 6796 consensus_meta.cc:287] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.564s user 0.020s sys 0.000s
I20260806 03:52:29.271584 6861 raft_consensus.cc:3032] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:29.422374 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.422757 2332 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:29.423760 2575 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:29.427124 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2246) took 1120 ms. Trace:
I20260806 03:52:29.382754 6800 raft_consensus.cc:686] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:29.395818 6780 raft_consensus.cc:729] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:29.280953 6841 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.438277 2478 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:52:29.425729 6688 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.311240 6803 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.437805 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:29.428470 6802 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.910s user 0.006s sys 0.008s
I20260806 03:52:29.439337 2575 raft_consensus.cc:595] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:29.397218 6836 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:29.432379 6815 consensus_meta.cc:287] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.733s user 0.017s sys 0.002s
W20260806 03:52:29.440820 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58219.3us
I20260806 03:52:29.424643 6857 raft_consensus.cc:729] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:29.441097 2476 rpcz_store.cc:276] 0806 03:52:28.306349 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.662567 (+356218us) service_pool.cc:227] Handling call
0806 03:52:28.697309 (+ 34742us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:28.697599 (+ 290us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:52:28.759617 (+ 62018us) log.cc:842] Serialized 14 byte log entry
0806 03:52:29.426547 (+666930us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:29.426641 (+ 94us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:29.426714 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:29.426799 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:29.427044 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37861,"prepare.queue_time_us":63560,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":65,"thread_start_us":666378,"threads_started":1}
I20260806 03:52:29.454231 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:29.465914 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.436369 6489 consensus_queue.cc:260] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.425431 6899 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
W20260806 03:52:29.512261 6890 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.084s user 0.000s sys 0.001s
I20260806 03:52:29.454713 6212 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:29.593628 6800 consensus_queue.cc:237] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.677487 6212 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.512283 6490 raft_consensus.cc:3032] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:29.437467 6772 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.437806 6901 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:29.512953 6858 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s user 0.010s sys 0.008s
W20260806 03:52:29.682159 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 233021us
I20260806 03:52:29.616734 6780 consensus_queue.cc:260] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.617220 6384 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.453881 6905 consensus_queue.cc:1046] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.260s
I20260806 03:52:29.454136 6902 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:29.585362 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 406) took 1933 ms. Trace:
I20260806 03:52:29.593811 6907 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:29.617519 6627 consensus_meta.cc:287] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.165s user 0.015s sys 0.000s
I20260806 03:52:29.466102 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.604394 6842 raft_consensus.cc:3032] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
W20260806 03:52:29.682528 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 159052us
W20260806 03:52:29.693022 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 273334us
I20260806 03:52:29.513196 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.713515 2478 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:29.713589 3545 leader_election.cc:334] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Timed out: RequestConsensusVote RPC to 127.0.183.3:46407 timed out after 1.692s (SENT)
I20260806 03:52:29.714177 3545 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:29.511971 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:29.531162 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.682734 6818 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.685124 6901 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:29.702431 6858 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.053s user 0.018s sys 0.011s
I20260806 03:52:29.689419 6780 raft_consensus.cc:3037] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:29.616285 6698 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:29.483592 6796 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:29.745297 6796 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.748150 6901 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.685308 6899 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.749002 6212 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:29.749598 6899 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:29.684839 2479 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:29.695386 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.696408 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.709870 2483 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:52:29.512785 6897 log.cc:924] Time spent T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.230s user 0.000s sys 0.000s
I20260806 03:52:29.717252 6842 raft_consensus.cc:729] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:29.719122 6907 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.683482 6688 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.721814 6893 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.685468 6772 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:52:29.688167 2575 raft_consensus.cc:603] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:29.716089 2583 raft_consensus.cc:2379] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:52:29.717131 3591 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s user 0.013s sys 0.005s
W20260806 03:52:29.453601 6525 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.110s user 0.019s sys 0.000s
I20260806 03:52:29.703575 2477 rpcz_store.cc:276] 0806 03:52:27.651673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:27.738038 (+ 86365us) service_pool.cc:227] Handling call
0806 03:52:27.759026 (+ 20988us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:27.782431 (+ 23405us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:27.795713 (+ 13282us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:27.796299 (+ 586us) log.cc:842] Serialized 26 byte log entry
0806 03:52:28.020997 (+224698us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:28.296592 (+275595us) log.cc:842] Serialized 14 byte log entry
0806 03:52:28.297060 (+ 468us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:28.297174 (+ 114us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:29.585074 (+1287900us) raft_consensus.cc:1671] finished
0806 03:52:29.585106 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:29.585284 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":349247,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":17916,"raft.queue_time_us":19828,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":17428,"thread_start_us":405791,"threads_started":2,"wal-append.queue_time_us":569997}
I20260806 03:52:29.511914 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.679575 6490 raft_consensus.cc:729] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:29.728904 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:29.757615 6915 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.781312 6915 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:29.782253 6672 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.264s user 0.012s sys 0.004s
I20260806 03:52:29.782917 6672 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.757969 6843 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:29.704648 6891 log.cc:924] Time spent T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.636s user 0.002s sys 0.000s
I20260806 03:52:29.724347 6802 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:52:29.766984 2484 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 9: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:29.768262 6797 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:29.768424 6850 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:29.769050 6855 consensus_meta.cc:287] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.429s user 0.008s sys 0.008s
I20260806 03:52:29.690423 6905 raft_consensus.cc:3032] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:29.787783 6855 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:29.741288 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.755081 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:29.454767 6809 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.763255 6845 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:29.837895 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:29.838652 2379 raft_consensus.cc:1226] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 3. Current term is 4. Ops: []
I20260806 03:52:29.784993 6841 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.787039 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 440) took 1364 ms. Trace:
I20260806 03:52:29.719484 6815 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.775390 2370 raft_consensus.cc:2396] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
W20260806 03:52:29.703130 2332 leader_election.cc:334] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.662s (SENT)
I20260806 03:52:29.777990 2481 raft_consensus.cc:2454] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:29.840830 2332 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:29.438349 6861 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:29.843115 6861 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.844486 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.844695 6861 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:29.808980 6803 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:29.774533 3591 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s user 0.014s sys 0.005s
I20260806 03:52:29.846102 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.771839 6688 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.769230 6907 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.846971 2478 raft_consensus.cc:2379] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 9: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:29.779505 6553 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.201s user 0.013s sys 0.000s
I20260806 03:52:29.764616 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2331) took 1474 ms. Trace:
I20260806 03:52:29.847651 6553 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:29.848652 2581 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:29.847718 2374 rpcz_store.cc:276] 0806 03:52:28.290540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.290669 (+ 129us) service_pool.cc:227] Handling call
0806 03:52:28.291071 (+ 402us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:28.444660 (+153589us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:28.444694 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:28.463291 (+ 18597us) log.cc:842] Serialized 26 byte log entry
0806 03:52:28.742593 (+279302us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:28.742642 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:28.970714 (+228072us) spinlock_profiling.cc:243] Waited 153 ms on lock 0x7b54001072d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:28.970804 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:29.764124 (+793320us) raft_consensus.cc:1671] finished
0806 03:52:29.764162 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:29.764519 (+ 357us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":162,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":12657,"spinlock_wait_cycles":352141184,"thread_start_us":229215,"threads_started":1,"wal-append.queue_time_us":670911}
I20260806 03:52:29.773967 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:29.795097 2474 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:29.850546 2575 raft_consensus.cc:2379] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:29.851769 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:29.852677 2575 raft_consensus.cc:2362] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6. Re-sending same reply.
I20260806 03:52:29.853834 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:29.777654 3586 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 9: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:29.777107 6490 consensus_queue.cc:260] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.788278 6905 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
W20260806 03:52:29.805451 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 79085.8us
I20260806 03:52:29.860008 6794 raft_consensus.cc:3032] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:29.863416 6794 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:29.864282 6904 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.828132 6796 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:29.864868 6904 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.866282 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 444) took 1336 ms. Trace:
W20260806 03:52:29.773103 6373 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.281s user 0.012s sys 0.008s
I20260806 03:52:29.866981 2370 rpcz_store.cc:276] 0806 03:52:28.530405 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.534020 (+ 3615us) service_pool.cc:227] Handling call
0806 03:52:29.865955 (+1331935us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003327d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:29.866185 (+ 230us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2783368064}
I20260806 03:52:29.835611 2477 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:52:29.837039 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:29.887782 2578 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 9
I20260806 03:52:29.839228 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.754565 6836 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.843544 2436 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:29.845381 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.867247 6373 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:29.862689 6895 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.477s user 0.004s sys 0.000s
I20260806 03:52:29.834842 2368 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:29.770220 6893 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.870992 6815 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:29.860574 6905 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.871658 6794 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:29.861732 6912 log.cc:924] Time spent T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.409s user 0.000s sys 0.001s
W20260806 03:52:29.780599 6854 consensus_meta.cc:287] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.241s user 0.016s sys 0.000s
I20260806 03:52:29.775213 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:29.839838 2372 rpcz_store.cc:276] 0806 03:52:28.422905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.424324 (+ 1419us) service_pool.cc:227] Handling call
0806 03:52:28.436395 (+ 12071us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:28.436774 (+ 379us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:28.436802 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:28.480317 (+ 43515us) log.cc:842] Serialized 487 byte log entry
0806 03:52:28.700895 (+220578us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:28.700939 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:28.701020 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:29.786705 (+1085685us) raft_consensus.cc:1671] finished
0806 03:52:29.786739 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:29.786943 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":144103,"threads_started":1,"wal-append.queue_time_us":382057}
I20260806 03:52:29.880831 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:29.894665 2376 raft_consensus.cc:1261] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 03:52:29.896226 2373 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:29.896672 6803 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260806 03:52:29.896746 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 193662us
I20260806 03:52:29.936110 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:29.875613 6765 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.898526 6793 raft_consensus.cc:3032] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:29.878336 6818 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.902455 3591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36468 (request call id 468) took 1140 ms. Trace:
I20260806 03:52:29.908601 6672 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 11 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:29.783776 6915 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:29.909878 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:29.925104 6688 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.765460 6842 consensus_queue.cc:260] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.920539 2476 raft_consensus.cc:2379] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:29.918577 3542 leader_election.cc:334] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.942s (SENT)
I20260806 03:52:29.921726 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:29.921523 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:29.923499 2482 raft_consensus.cc:1226] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 3. Current term is 4. Ops: []
W20260806 03:52:29.925683 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 82257.8us
I20260806 03:52:29.925777 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:29.957266 2369 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:29.937774 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.957984 2436 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:29.958250 2481 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:52:29.937518 2577 raft_consensus.cc:1226] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260806 03:52:29.958878 6934 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:29.888892 2483 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
W20260806 03:52:29.936851 2332 leader_election.cc:334] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.141s (SENT)
I20260806 03:52:29.960139 2332 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:29.959432 2436 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:29.944907 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:29.961404 2474 raft_consensus.cc:1261] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260806 03:52:29.942524 2368 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 2.
I20260806 03:52:29.924921 6907 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:29.689566 6384 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.951617 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:29.951643 2375 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:29.952857 2479 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:29.953680 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:29.966724 2374 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:29.953866 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:29.954553 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:29.929123 6851 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.957252 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:29.959403 2534 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:29.941304 3591 rpcz_store.cc:276] 0806 03:52:28.762219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.762343 (+ 124us) service_pool.cc:227] Handling call
0806 03:52:28.905566 (+143223us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:28.914430 (+ 8864us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:28.914479 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:28.963501 (+ 49022us) log.cc:842] Serialized 26 byte log entry
0806 03:52:29.846174 (+882673us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:29.846225 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:29.846302 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:29.902160 (+ 55858us) raft_consensus.cc:1671] finished
0806 03:52:29.902192 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:29.902376 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34746,"raft.queue_time_us":417917,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":18459,"thread_start_us":750446,"threads_started":1,"wal-append.queue_time_us":844530}
W20260806 03:52:29.945897 6799 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.008s sys 0.027s
W20260806 03:52:29.969405 6799 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.584s user 0.009s sys 0.028s
I20260806 03:52:29.970275 6909 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260806 03:52:29.938149 6918 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.972761 6918 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.975003 6918 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.965260 6904 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:29.976351 6855 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:29.976677 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:29.951920 6818 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.980340 6818 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:29.983337 6384 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:29.984064 6926 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.984896 2436 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:29.956511 3542 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:29.942642 6793 raft_consensus.cc:729] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:29.988605 6840 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:29.989108 6793 consensus_queue.cc:260] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.989748 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:29.990347 6793 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:29.991372 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:29.964708 6794 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:29.940863 6765 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:29.959775 6796 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260806 03:52:29.997946 6765 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:29.998055 6672 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 03:52:29.952220 6803 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:29.998757 6851 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.999207 6803 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.001588 6803 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.001775 6851 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:29.934330 6854 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.002460 2575 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:52:30.004174 6854 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:30.005077 2576 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:29.970446 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:29.984652 6926 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.015170 6907 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260806 03:52:29.986104 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 67590.2us
I20260806 03:52:29.949826 6905 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:30.016644 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.017450 2369 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
W20260806 03:52:30.018812 6825 consensus_meta.cc:287] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.869s user 0.008s sys 0.004s
I20260806 03:52:30.019588 6672 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:29.713778 6857 consensus_queue.cc:260] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.020141 6672 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:29.961823 2477 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:29.967096 6934 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:29.962594 6836 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.021651 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:30.021895 6836 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.022456 3585 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:29.960578 6688 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.011422 2370 raft_consensus.cc:1261] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260806 03:52:30.015686 6926 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:30.024094 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:30.024590 2376 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:30.026901 2578 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
W20260806 03:52:29.970494 6913 log.cc:924] Time spent T 00000000000000000000000000000000 P e1fd51235035487c913ffce700debedb: Append to log took a long time: real 0.182s user 0.000s sys 0.001s
I20260806 03:52:30.029332 2331 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.026947 2375 raft_consensus.cc:1261] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:30.030117 3585 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 9: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:29.975780 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:30.030453 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.032507 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:30.029335 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 44553.7us
I20260806 03:52:29.986313 6842 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:30.041972 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:30.044987 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 85683.3us
I20260806 03:52:30.054908 2579 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260806 03:52:30.056154 2578 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:30.056516 3545 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:30.045703 2332 leader_election.cc:334] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.676s (SENT)
I20260806 03:52:30.047020 6926 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 03:52:29.966097 6698 consensus_queue.cc:1159] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "dff94100ffc5485f92cbe6a85addf0b2" responder_term: 7 status { last_received { term: 6 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:52:30.048264 6855 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.049642 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.060667 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.056609 2534 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.057435 2332 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.045054 2436 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.061234 2583 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:30.028864 6825 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.064648 2579 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:30.036361 6836 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.075513 3545 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.068331 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.070855 2578 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:30.071811 2534 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.076531 2370 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:30.073630 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:30.060592 6783 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 03:52:30.075302 2436 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.051381 6803 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:30.085835 6645 raft_consensus.cc:3032] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:30.086292 6645 raft_consensus.cc:729] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:30.087080 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.087873 2584 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:30.089301 6928 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:30.090133 6862 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:30.090536 6862 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.090839 6862 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:30.055112 6851 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:30.105101 6851 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.107568 6851 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:30.059361 6926 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:30.108393 6926 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:30.113149 6809 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P cf6042e85bef47c4aac9d064d91501d1 [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:30.115404 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:30.117892 6801 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 11 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:30.117308 2368 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:30.118664 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:30.121214 6919 raft_consensus.cc:3032] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:30.121665 6919 raft_consensus.cc:729] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:30.122583 6919 consensus_queue.cc:260] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.123243 6893 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.075109 6839 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:30.124647 6839 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.125142 6839 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:30.080564 2436 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.067509 6904 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260806 03:52:30.095201 2579 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:30.127622 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2326) took 1340 ms. Trace:
I20260806 03:52:30.086558 6802 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:30.128089 2480 rpcz_store.cc:276] 0806 03:52:28.787270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.913074 (+125804us) service_pool.cc:227] Handling call
0806 03:52:30.127374 (+1214300us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b540019f354. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:30.127516 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2263274368}
I20260806 03:52:30.128566 6802 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.128794 2484 raft_consensus.cc:1226] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:30.129038 6802 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:30.078403 6825 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.074823 6909 consensus_queue.cc:1046] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.022s
I20260806 03:52:30.096879 6896 consensus_queue.cc:1046] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260806 03:52:30.070546 6861 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:30.132628 6825 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.065299 6855 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.133268 6818 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.133891 6818 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.135190 6855 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.099232 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 439) took 1676 ms. Trace:
I20260806 03:52:30.103778 6841 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:30.104280 6803 raft_consensus.cc:686] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:30.087255 6645 consensus_queue.cc:260] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.123356 6800 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:30.092869 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.081687 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:30.140906 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48069.4us
W20260806 03:52:30.168718 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 86962us
W20260806 03:52:30.169940 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 89480.4us
I20260806 03:52:30.137302 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.176218 6928 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:30.176437 2572 raft_consensus.cc:2379] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:30.167608 2379 rpcz_store.cc:276] 0806 03:52:28.422439 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.423511 (+ 1072us) service_pool.cc:227] Handling call
0806 03:52:28.423964 (+ 453us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:29.839054 (+1415090us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:30.098888 (+259834us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b540026fa54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:30.099143 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3253083392}
I20260806 03:52:30.178020 2583 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 9: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:30.179610 2375 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
W20260806 03:52:30.180727 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 55930.5us
I20260806 03:52:30.181699 2436 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.182628 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.123881 6919 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:30.187506 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:30.190274 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.190896 2484 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Advancing to term 11
I20260806 03:52:30.190670 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.131884 6672 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:30.194984 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.191690 2474 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:30.144603 6489 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:30.191066 2584 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:30.149155 6770 raft_consensus.cc:3032] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:30.139665 6785 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.135923 6818 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.145318 6688 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:30.198275 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 131603us
I20260806 03:52:30.201824 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:30.234120 2377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 433) took 1889 ms (client timeout 2175 ms). Trace:
I20260806 03:52:30.234673 2572 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
W20260806 03:52:30.234750 2377 rpcz_store.cc:269] 0806 03:52:28.345135 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:28.345318 (+ 183us) service_pool.cc:227] Handling call
0806 03:52:30.233875 (+1888557us) spinlock_profiling.cc:243] Waited 777 ms on lock 0x7b5400438fd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:30.234048 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1786355584}
W20260806 03:52:30.234606 2332 leader_election.cc:334] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.698s (SENT)
I20260806 03:52:30.204912 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.204789 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.236177 2332 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.205664 6857 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:30.205873 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:30.206538 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.240060 2574 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:30.207350 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:30.207988 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 113707us
I20260806 03:52:30.240981 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.209980 2538 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.211014 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.177006 6928 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.167788 6841 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:30.199522 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.215044 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:30.222167 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 49217.7us
I20260806 03:52:30.250046 2480 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:30.141216 5510 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260806 03:52:30.251755 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.252971 2482 raft_consensus.cc:3032] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:30.241894 2484 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 11.
I20260806 03:52:30.253556 2482 raft_consensus.cc:729] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:30.255026 2483 raft_consensus.cc:1226] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:30.192097 6917 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.265185 2481 raft_consensus.cc:1226] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:30.266639 2481 raft_consensus.cc:1226] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
W20260806 03:52:30.267259 2332 leader_election.cc:334] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.792s (SENT)
I20260806 03:52:30.267814 2332 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:30.268235 6815 raft_consensus.cc:2781] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:52:30.268723 6815 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.269096 6815 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:52:30.269382 2332 leader_election.cc:334] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.895s (SENT)
I20260806 03:52:30.269855 2332 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.270956 6831 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:30.213228 6855 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:30.198720 6934 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.253031 3547 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:30.227330 6770 raft_consensus.cc:729] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:30.242573 2477 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:30.245503 6928 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:30.287258 3547 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.262224 2476 raft_consensus.cc:1261] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260806 03:52:30.299718 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.263156 6698 consensus_queue.cc:1035] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:52:30.254901 6841 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:30.264014 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.254467 2482 consensus_queue.cc:260] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.272451 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.236747 2368 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 4.
I20260806 03:52:30.250360 3588 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
W20260806 03:52:30.274544 2332 leader_election.cc:334] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.962s (SENT)
I20260806 03:52:30.277879 6373 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:30.271497 6831 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:30.247217 3587 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:30.276415 6785 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.267765 6920 raft_consensus.cc:2726] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:30.277948 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.254849 2474 raft_consensus.cc:2932] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 3 to 5, VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:30.281885 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.288239 2534 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.296598 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.281353 3585 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 11: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:30.254143 6800 raft_consensus.cc:2741] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260806 03:52:30.256204 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.300477 2370 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:30.304426 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:30.307927 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:30.302445 2572 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 9: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:30.301417 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:30.308439 2478 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:30.298817 6855 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:30.308614 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.314332 2332 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.381678 2477 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
W20260806 03:52:30.382217 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 111080us
W20260806 03:52:30.383184 2332 leader_election.cc:334] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.008s (SENT)
I20260806 03:52:30.275069 2484 raft_consensus.cc:1226] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:30.298652 3584 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:30.315443 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.325945 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.328763 2534 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.318955 6915 raft_consensus.cc:3032] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:30.331424 2483 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:30.412772 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.413436 2483 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:30.279237 6688 consensus_queue.cc:1046] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260806 03:52:30.356429 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.349893 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.357316 6802 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.346697 6909 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:30.359929 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.364146 6816 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:30.309891 3547 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:30.389137 6899 raft_consensus.cc:2726] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:30.299157 6934 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.368378 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.370500 3591 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:30.331395 6785 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.373768 3586 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:30.377363 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.265583 6917 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.385310 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.430253 2481 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:30.431912 2484 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:30.383731 2332 leader_election.cc:302] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.313040 6904 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:30.409397 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 80709.8us
I20260806 03:52:30.398979 2538 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:30.410343 2578 raft_consensus.cc:2354] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:30.434937 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.316448 6818 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.411166 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.410857 2476 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:30.417153 6893 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:30.380088 6799 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:30.419874 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.379861 6794 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Leader election won for term 11
I20260806 03:52:30.416234 2478 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
W20260806 03:52:30.425320 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 115524us
I20260806 03:52:30.366124 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.397468 6920 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:30.372174 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.412215 6894 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:52:30.303257 6906 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:30.437932 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:30.438719 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 55587.4us
I20260806 03:52:30.438733 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.439987 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.436086 6809 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:37794 (request call id 3) took 11485 ms. Trace:
I20260806 03:52:30.420192 6915 raft_consensus.cc:729] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:30.397105 6855 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.444808 2574 raft_consensus.cc:2354] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5.
I20260806 03:52:30.429572 6783 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:30.293524 6803 consensus_queue.cc:237] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:30.445317 2483 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:30.446048 2436 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.446919 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:30.447165 2474 raft_consensus.cc:1226] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:30.414018 6793 consensus_queue.cc:1035] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.026s
I20260806 03:52:30.448329 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:30.449872 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 51074.2us
I20260806 03:52:30.450335 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.449213 2534 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.453984 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:30.454078 2332 leader_election.cc:334] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.080s (SENT)
I20260806 03:52:30.448172 6920 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.453318 6794 raft_consensus.cc:686] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [term 11 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:30.461985 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.468765 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:30.437134 6904 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.454988 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.451984 6926 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.435925 6818 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.478852 6917 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.480260 2583 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:30.459038 6809 rpcz_store.cc:276] 0806 03:52:18.964353 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:18.992973 (+ 28620us) service_pool.cc:227] Handling call
0806 03:52:21.493095 (+2500122us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b58007826b0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7ca40f63 00007f0d7caa4aba 00007f0d7cac0a7a 00007f0d7cac1ada 00007f0d7cac0c8b 00007f0d7cacc084 00007f0d7cacb743 00007f0d7cac3c1a 00007f0d81a076d9 00007f0d7b0f1184
0806 03:52:22.021896 (+528801us) log.cc:842] Serialized 487 byte log entry
0806 03:52:30.435908 (+8414012us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":553047,"raft.run_cpu_time_us":16625,"raft.run_wall_time_us":895512,"spinlock_wait_cycles":926624128,"thread_start_us":960061,"threads_started":2,"wal-append.queue_time_us":744570,"wal-append.run_cpu_time_us":2185,"wal-append.run_wall_time_us":1065655}
I20260806 03:52:30.578498 6785 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.580145 6918 raft_consensus.cc:3032] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:30.580665 6918 raft_consensus.cc:729] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:30.580868 2296 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet dfa9a0499fc04e10aefa0602a7adbbb1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260806 03:52:30.581696 6918 consensus_queue.cc:260] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.583256 6918 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:30.587872 2482 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:30.589628 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.457929 5510 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:30.589972 6855 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.592747 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.593623 6818 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.594127 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.422472 6770 consensus_queue.cc:260] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.597551 2481 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:30.597708 3545 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.487581 6896 raft_consensus.cc:3032] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:30.454025 2484 raft_consensus.cc:2379] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 11: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:30.443292 6899 raft_consensus.cc:2726] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:30.429382 6816 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:30.604802 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 150763us
I20260806 03:52:30.594913 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:30.609427 2332 leader_election.cc:334] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.097s (SENT)
W20260806 03:52:30.596659 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 147534us
I20260806 03:52:30.453568 6906 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.599181 2478 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:30.599639 3545 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.600684 2483 raft_consensus.cc:1261] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260806 03:52:30.601261 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 154384us
I20260806 03:52:30.444743 6893 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.607745 2476 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:30.608820 2482 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:52:30.609263 2538 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:30.598701 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 152783us
W20260806 03:52:30.595646 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 285507us
W20260806 03:52:30.606261 6862 consensus_meta.cc:287] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.515s user 0.009s sys 0.004s
I20260806 03:52:30.610185 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.611344 2484 raft_consensus.cc:1261] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:30.613921 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.616374 3545 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.617259 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:30.639981 2481 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 9
I20260806 03:52:30.605516 6688 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:30.641682 6688 raft_consensus.cc:686] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:30.598475 6796 consensus_queue.cc:1159] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "608e57d72aae496aaf8613622153aa0c" responder_term: 6 status { last_received { term: 5 index: 3 } last_committed_idx: 3 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:52:30.642357 6794 consensus_queue.cc:237] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.642741 6688 consensus_queue.cc:237] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.644260 6770 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:30.646142 6816 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:30.647236 6920 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:30.602267 6934 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.626524 2436 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.449713 6799 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.612480 6627 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:30.630589 3591 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 03:52:30.668648 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.638769 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.556730 6831 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.609829 2332 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:30.650493 2534 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.623553 2579 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:30.651223 2538 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.602636 6896 raft_consensus.cc:729] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:30.653931 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.678870 2480 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:30.656347 2436 leader_election.cc:398] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:30.578416 6915 consensus_queue.cc:260] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.656005 2331 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.618103 6906 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:30.612111 6783 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.694653 6899 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:30.694875 6783 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:30.604193 5510 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:30.661007 6861 consensus_queue.cc:1046] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260806 03:52:30.695182 6899 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.703173 6899 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:30.691380 6920 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:30.619585 6802 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.622489 6917 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 03:52:30.707777 6861 raft_consensus.cc:3032] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:30.708218 6861 raft_consensus.cc:729] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:30.674767 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.661341 6809 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.015s
I20260806 03:52:30.709275 6904 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:30.676132 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.709074 6861 consensus_queue.cc:260] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.710762 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.711179 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:30.676705 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 67325.5us
W20260806 03:52:30.712755 2332 leader_election.cc:334] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.102s (SENT)
I20260806 03:52:30.638176 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:30.612847 6785 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:30.673573 2481 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:30.688167 2331 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.654182 6905 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:30.690779 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:30.729624 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 73414.8us
I20260806 03:52:30.632783 5828 consensus_queue.cc:1035] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260806 03:52:30.730446 2583 raft_consensus.cc:2354] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:30.730660 2436 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.731628 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.732384 2436 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.619715 6854 consensus_queue.cc:1035] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
W20260806 03:52:30.713968 2332 leader_election.cc:334] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.130s (SENT)
I20260806 03:52:30.677526 2534 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.661049 6800 raft_consensus.cc:2726] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:30.714679 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:30.726259 6945 log.cc:924] Time spent T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260806 03:52:30.664188 6627 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.710486 6861 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:30.727479 2331 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.727958 3547 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:30.734328 2436 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.739758 2332 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.667593 6841 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.742978 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.660791 6851 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:52:30.727135 6919 consensus_meta.cc:287] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.518s user 0.008s sys 0.006s
I20260806 03:52:30.692584 6816 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:30.690692 6553 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:30.719350 5510 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:30.745826 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 68360.6us
I20260806 03:52:30.702339 6893 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:30.750800 2332 leader_election.cc:334] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.139s (SENT)
I20260806 03:52:30.729410 6785 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.654561 6862 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:30.794484 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 66621.3us
W20260806 03:52:30.795099 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 44350.4us
W20260806 03:52:30.796280 2332 leader_election.cc:334] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.147s (SENT)
I20260806 03:52:30.796804 2332 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:30.797483 2534 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:30.733152 6920 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:30.688426 6831 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.733233 6907 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:30.739902 6934 raft_consensus.cc:2726] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:30.811017 2534 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:30.745393 6915 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:30.816510 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.802029 2579 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:30.841559 2370 raft_consensus.cc:1226] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:30.842020 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.745204 6904 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:30.800391 6785 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:30.861515 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.862874 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:30.864665 6489 consensus_meta.cc:287] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.648s user 0.007s sys 0.007s
I20260806 03:52:30.753952 6627 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:30.842389 6013 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election won for term 9
W20260806 03:52:30.868446 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 134189us
I20260806 03:52:30.869004 6013 raft_consensus.cc:686] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:30.869663 2436 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.816344 6857 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:30.870909 6857 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.871309 6857 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:30.843176 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.872363 6904 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.817499 6900 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:30.875186 6900 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.800688 5510 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:30.877321 6900 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:30.880832 6841 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:30.881539 6841 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:30.882362 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:30.802064 6672 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:30.884902 6672 raft_consensus.cc:686] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 9 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:30.887281 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.888181 2579 raft_consensus.cc:2354] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:30.752177 6926 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:30.816641 6851 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:30.825100 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:30.817057 6816 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.860002 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:30.860368 6831 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.697247 6896 consensus_queue.cc:260] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:30.906570 2436 leader_election.cc:398] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5.
I20260806 03:52:30.861001 6862 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643)
I20260806 03:52:30.861137 2376 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 03:52:30.870422 6013 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.801504 6800 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:30.800673 6855 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:30.804203 6920 raft_consensus.cc:482] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:30.858968 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:30.816856 6553 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:30.895478 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 47928.1us
I20260806 03:52:30.818619 6836 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:30.898492 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:30.743341 6928 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:30.902222 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 105978us
I20260806 03:52:30.910233 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:30.912748 2574 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:30.875550 6904 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.914608 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.964242 3584 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:30.965231 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:30.890655 6839 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.765s user 0.015s sys 0.003s
I20260806 03:52:30.886181 6672 consensus_queue.cc:237] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:30.968986 3542 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.916739 3585 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 9: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:30.930017 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:30.896704 6851 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:30.931896 6490 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:30.946372 2369 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:30.899438 6816 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:30.951511 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 45099.7us
I20260806 03:52:30.989400 2436 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.989717 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:30.990494 2579 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:30.990721 2436 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.898840 6688 raft_consensus.cc:2726] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:30.855011 2584 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 9.
I20260806 03:52:31.000272 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:30.971230 6896 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:31.007443 2436 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:30.974663 2368 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:30.970199 2331 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:30.918524 6855 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:30.946576 2332 leader_election.cc:334] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.654s (SENT)
I20260806 03:52:31.019088 2332 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:30.985004 2479 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:30.938741 6920 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:30.954357 6842 consensus_meta.cc:287] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.896s user 0.014s sys 0.000s
I20260806 03:52:30.982595 6802 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:30.946338 6553 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:31.012240 6934 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:31.020574 2436 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.018366 5510 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.022553 6855 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:30.992514 6783 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:31.022825 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:31.029470 6627 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:31.075553 2379 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:52:31.023162 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.076412 2369 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
W20260806 03:52:31.036064 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 89540.1us
I20260806 03:52:31.036629 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:31.078624 2584 raft_consensus.cc:2379] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:31.004284 6800 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:31.032143 6904 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:31.040100 2579 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:31.086663 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:31.008873 6799 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:30.976305 6839 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.080456 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.037212 2575 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:31.037330 6907 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:31.114576 6928 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:31.090447 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:31.122603 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:31.094123 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:31.094498 2376 raft_consensus.cc:1261] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:52:31.095146 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.082020 6899 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.096457 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.044327 6836 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:31.100961 2576 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:31.102349 6645 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:31.103745 6800 raft_consensus.cc:686] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:31.100888 6958 log.cc:924] Time spent T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260806 03:52:31.131928 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:31.161976 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:31.163209 2584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:31.131544 6907 raft_consensus.cc:686] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:31.164973 6907 consensus_queue.cc:237] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.166191 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.091233 6904 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:31.168190 3588 raft_consensus.cc:2354] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6.
I20260806 03:52:31.168315 6904 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:31.168840 6800 consensus_queue.cc:237] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.171308 5510 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:31.132622 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.136935 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:31.176267 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.142469 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:31.176668 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0751082990318537s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
W20260806 03:52:31.177163 2332 connection.cc:232] RPC call timeout handler was delayed by -1.0346082989126444s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
I20260806 03:52:31.148658 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.150230 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.150951 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:31.163805 2584 raft_consensus.cc:595] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:31.134944 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:31.152060 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 470909us
I20260806 03:52:31.182104 2538 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:31.183856 2332 leader_election.cc:334] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.221s (SENT)
I20260806 03:52:31.183856 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:31.184371 5510 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:31.184528 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.184880 5510 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:31.185284 5510 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:31.185395 6929 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:31.186324 6841 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:31.159520 2581 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
W20260806 03:52:31.186715 2332 leader_election.cc:334] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.763s (SENT)
I20260806 03:52:31.188580 2538 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:31.152627 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 110987us
I20260806 03:52:31.171813 6785 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:31.153363 2379 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
W20260806 03:52:31.153635 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 133177us
I20260806 03:52:31.081395 6934 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:31.192533 2575 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 10: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:31.157789 2583 raft_consensus.cc:2354] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:31.167604 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.194269 2436 leader_election.cc:398] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:31.189878 2373 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 10: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:31.192588 2474 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260806 03:52:31.131253 6698 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:52:31.151371 6770 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.507s user 0.006s sys 0.003s
W20260806 03:52:31.152822 6939 log.cc:924] Time spent T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
W20260806 03:52:31.176255 6815 consensus_meta.cc:287] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.907s user 0.014s sys 0.002s
I20260806 03:52:31.198968 2575 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 5. Current term is 6. Ops: [5.8-5.8]
I20260806 03:52:31.187606 2577 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 3.
I20260806 03:52:31.201335 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:31.203756 2577 raft_consensus.cc:1226] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260806 03:52:31.206786 2379 raft_consensus.cc:1261] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260806 03:52:31.211498 2376 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:52:31.214277 2577 raft_consensus.cc:1261] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260806 03:52:31.188652 6836 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:31.224270 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:31.226487 3585 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260806 03:52:31.226727 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:31.227412 2482 raft_consensus.cc:3037] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:31.228468 6785 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:31.229897 6783 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:31.219728 2574 raft_consensus.cc:1226] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:52:31.241076 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:31.242484 2484 raft_consensus.cc:2396] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:52:31.251472 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:31.251657 2584 raft_consensus.cc:1226] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260806 03:52:31.252640 2484 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:31.203840 6384 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260806 03:52:31.268064 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 73917.2us
I20260806 03:52:31.270382 2581 raft_consensus.cc:1261] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:52:31.258450 2482 raft_consensus.cc:2454] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:31.276690 2436 leader_election.cc:398] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
W20260806 03:52:31.274140 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 53880.5us
I20260806 03:52:31.277024 2436 leader_election.cc:401] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 03:52:31.278158 2436 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.286713 2436 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:31.290736 6918 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:31.291486 6918 raft_consensus.cc:686] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:31.293304 6918 consensus_queue.cc:237] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.298341 2436 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.302174 6920 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.303692 6799 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:31.305167 2436 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:31.306300 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:31.336108 2374 raft_consensus.cc:3037] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:52:31.308393 6627 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:31.341638 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:31.342624 2368 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:31.308519 6899 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:31.313815 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:31.330654 2534 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:31.352613 2476 raft_consensus.cc:3037] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:31.336848 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:31.202422 6926 consensus_queue.cc:1035] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.072s
I20260806 03:52:31.201956 6672 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.072s
I20260806 03:52:31.313158 6384 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:31.346438 2369 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 8: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:31.349836 2332 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.351540 2484 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:31.357213 2574 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:31.208201 6815 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.352705 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:31.359843 2534 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:31.219377 6894 consensus_queue.cc:1035] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.119s
I20260806 03:52:31.313632 6770 raft_consensus.cc:2726] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:31.340512 2436 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:31.404654 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:31.416273 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 177528us
I20260806 03:52:31.416736 2477 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
W20260806 03:52:31.418462 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 78349.7us
I20260806 03:52:31.422180 2476 raft_consensus.cc:2454] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 10.
I20260806 03:52:31.336203 6928 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:31.268229 6765 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.095s
I20260806 03:52:31.346122 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:31.229897 6904 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:31.428987 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 79253us
I20260806 03:52:31.438694 2332 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.373468 6384 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:31.429904 2368 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:31.376709 6893 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:31.430903 6841 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:31.428700 6688 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader election won for term 4
I20260806 03:52:31.493084 2477 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 5.
I20260806 03:52:31.447844 2574 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 10.
I20260806 03:52:31.443068 2332 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.532562 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:31.533485 2574 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 10: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:52:31.495596 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 173005us
W20260806 03:52:31.532617 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 89626.9us
I20260806 03:52:31.535298 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 365) took 1048 ms. Trace:
I20260806 03:52:31.535989 2332 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.535876 2581 rpcz_store.cc:276] 0806 03:52:30.486468 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.231571 (+745103us) service_pool.cc:227] Handling call
0806 03:52:31.270012 (+ 38441us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:31.275839 (+ 5827us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:31.535010 (+259171us) spinlock_profiling.cc:243] Waited 38.1 ms on lock 0x7b540024d6d0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:31.535212 (+ 202us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":87557376}
I20260806 03:52:31.449545 2436 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:31.438944 6793 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:31.494643 2374 raft_consensus.cc:2454] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
W20260806 03:52:31.473862 6967 log.cc:924] Time spent T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.166s user 0.000s sys 0.001s
I20260806 03:52:31.464828 3585 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:31.470355 6899 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:31.475145 6928 raft_consensus.cc:686] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:31.543347 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 117125us
I20260806 03:52:31.549602 2332 leader_election.cc:302] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:31.550359 3542 leader_election.cc:302] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:31.523370 6839 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:31.475723 6770 consensus_queue.cc:1035] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.074s
I20260806 03:52:31.557433 6926 raft_consensus.cc:2781] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260806 03:52:31.568507 2538 leader_election.cc:302] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:31.567809 2574 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb91ba465b9845129828f07c4d452970"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
W20260806 03:52:31.560442 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 213273us
I20260806 03:52:31.569343 2574 raft_consensus.cc:595] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
W20260806 03:52:31.549865 6857 consensus_meta.cc:287] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.678s user 0.008s sys 0.006s
I20260806 03:52:31.560192 2332 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:31.571309 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 121873us
I20260806 03:52:31.482704 6925 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260806 03:52:31.543358 6836 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:31.575939 2332 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.488217 6627 consensus_queue.cc:1046] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.070s
I20260806 03:52:31.491173 6893 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:31.556706 6785 raft_consensus.cc:2726] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 03:52:31.604959 6851 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.611s user 0.008s sys 0.004s
I20260806 03:52:31.564641 6688 raft_consensus.cc:686] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:31.621500 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 261769us
W20260806 03:52:31.620225 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 44371.9us
I20260806 03:52:31.579236 6907 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:31.573511 6926 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:31.553784 6841 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:31.578167 6934 raft_consensus.cc:2726] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:31.583736 6961 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:31.589777 6839 raft_consensus.cc:2781] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
W20260806 03:52:31.591435 6896 consensus_meta.cc:287] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.570s user 0.010s sys 0.003s
W20260806 03:52:31.649942 2480 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2360) took 1332 ms (client timeout 1591 ms). Trace:
W20260806 03:52:31.735338 2480 rpcz_store.cc:269] 0806 03:52:30.317291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.359581 (+ 42290us) service_pool.cc:227] Handling call
0806 03:52:31.649645 (+1290064us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b54002b1954. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:31.649826 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":583720320}
W20260806 03:52:31.651186 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 59099.9us
I20260806 03:52:31.622575 6836 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:31.618782 6920 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:31.584832 6985 raft_consensus.cc:2781] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader election won for term 6
W20260806 03:52:31.607919 6553 consensus_meta.cc:287] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.534s user 0.009s sys 0.004s
I20260806 03:52:31.736568 2538 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:31.659307 2534 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:31.633451 6861 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.838s user 0.013s sys 0.002s
I20260806 03:52:31.614455 6893 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:31.740700 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 170952us
W20260806 03:52:31.743942 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 166740us
I20260806 03:52:31.818473 2331 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:31.649240 6926 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:31.730443 6907 raft_consensus.cc:686] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:31.734979 6783 raft_consensus.cc:3032] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:31.609202 6770 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.208s
I20260806 03:52:31.585523 6857 raft_consensus.cc:504] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.827605 2375 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:31.631901 6851 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:31.838050 6816 consensus_meta.cc:287] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.845s user 0.009s sys 0.005s
W20260806 03:52:31.841387 6906 consensus_meta.cc:287] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.113s user 0.013s sys 0.002s
I20260806 03:52:31.652082 6785 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.170s
I20260806 03:52:31.826737 6920 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:52:31.828361 6815 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:31.844036 6855 consensus_meta.cc:287] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.757s user 0.010s sys 0.002s
I20260806 03:52:31.917680 2577 raft_consensus.cc:1226] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:31.835193 6783 raft_consensus.cc:729] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:31.833384 6861 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:31.926735 7004 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:31.931728 6698 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:31.932317 6974 log.cc:924] Time spent T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.345s user 0.000s sys 0.001s
I20260806 03:52:31.734975 6928 consensus_queue.cc:237] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.938194 3547 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:31.940135 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 312567us
I20260806 03:52:31.890614 6904 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:31.823626 6553 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:31.941854 2583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1134) took 1680 ms (client timeout 2154 ms). Trace:
I20260806 03:52:31.914943 6995 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:31.914036 6836 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:31.958271 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 221814us
I20260806 03:52:31.914172 6582 consensus_queue.cc:1035] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260806 03:52:31.960780 2332 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:31.960852 2583 rpcz_store.cc:269] 0806 03:52:30.252239 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.731222 (+478983us) service_pool.cc:227] Handling call
0806 03:52:31.932642 (+1201420us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b540013c4d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:31.932854 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":978043264}
I20260806 03:52:31.897346 6816 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.976617 2332 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.979245 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:31.980049 3586 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:31.892978 6906 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:31.997929 6553 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:31.938828 6920 raft_consensus.cc:686] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:31.978648 6904 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:31.999154 6841 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:31.995527 6997 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:31.966300 6855 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:31.994796 6490 consensus_meta.cc:287] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.999s user 0.015s sys 0.000s
I20260806 03:52:32.042537 6841 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:31.996248 6996 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 5 FOLLOWER]: Leader election won for term 5
I20260806 03:52:32.039604 6783 consensus_queue.cc:260] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:31.945925 6698 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:31.940490 6861 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.001606 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:31.981720 2332 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:31.939518 6977 raft_consensus.cc:2781] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:52:32.006084 2579 raft_consensus.cc:1226] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260806 03:52:31.997931 6816 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:32.005146 732 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260806 03:52:31.958321 6905 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:31.947486 6839 raft_consensus.cc:686] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:31.945075 6993 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:32.033257 3591 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:52:31.991386 3586 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:31.912058 6688 consensus_queue.cc:237] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.055330 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:32.052469 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:32.051492 2372 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:31.999781 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 364) took 1513 ms. Trace:
I20260806 03:52:31.981273 6836 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:31.972637 6995 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:31.950625 7004 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:31.959076 6907 consensus_queue.cc:237] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.195716 2477 raft_consensus.cc:2379] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 7: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:32.196913 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:32.155215 3590 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260806 03:52:32.135066 2572 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:32.198689 2478 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:32.120987 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 461795us
I20260806 03:52:32.127480 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:32.176704 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:32.200619 2331 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:32.044948 6384 consensus_meta.cc:287] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.521s user 0.012s sys 0.000s
I20260806 03:52:32.194083 2577 rpcz_store.cc:276] 0806 03:52:30.486136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.215046 (+728910us) service_pool.cc:227] Handling call
0806 03:52:31.215347 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:31.924828 (+709481us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:31.999500 (+ 74672us) spinlock_profiling.cc:243] Waited 702 ms on lock 0x7b540013b5d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:31.999708 (+ 208us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1614668672}
I20260806 03:52:32.197849 2477 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:32.207504 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:32.089375 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 432) took 1216 ms. Trace:
I20260806 03:52:32.209122 2481 rpcz_store.cc:276] 0806 03:52:30.873443 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.873637 (+ 194us) service_pool.cc:227] Handling call
0806 03:52:30.873978 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:30.921176 (+ 47198us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:30.921224 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:30.921280 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:30.921325 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:32.089096 (+1167771us) spinlock_profiling.cc:243] Waited 46.9 ms on lock 0x7b540019f854. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:32.089150 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:32.089299 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":107975040}
I20260806 03:52:32.043030 6934 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.063344 2436 leader_election.cc:398] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:32.051208 6841 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:32.050676 6899 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:32.052583 3584 raft_consensus.cc:1226] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:32.048228 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:32.096823 6553 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.050258 6855 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.052419 6996 raft_consensus.cc:686] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:32.238797 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:32.265731 2481 raft_consensus.cc:2354] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6.
I20260806 03:52:32.266990 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:32.053711 6783 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:32.239571 2534 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:32.242481 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:32.255929 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:32.192173 6985 raft_consensus.cc:686] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:32.166167 6857 leader_election.cc:288] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:32.047799 7003 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.216207 2583 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 6.
I20260806 03:52:32.198004 6851 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:32.214511 2436 leader_election.cc:401] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260806 03:52:32.056082 6997 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.260010 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:32.312235 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 330726us
I20260806 03:52:32.362938 2574 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:32.363180 2332 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:32.364590 2332 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:32.366669 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:32.212697 6993 raft_consensus.cc:686] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:32.315701 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:32.368098 2481 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:32.327666 2478 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
W20260806 03:52:32.336572 3547 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.141s (SENT) [suppressed 21 similar messages]
W20260806 03:52:32.357357 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 294166us
I20260806 03:52:32.196719 6770 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:32.373907 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 134470us
I20260806 03:52:32.372251 6953 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.376327 2538 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:32.048640 6920 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.205317 6905 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.311295 6855 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:32.236433 6934 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:32.359920 5510 consensus_meta.cc:287] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.174s user 0.017s sys 0.000s
I20260806 03:52:32.388599 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:32.415812 2481 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:32.373517 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:32.417641 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:32.419584 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 219053us
I20260806 03:52:32.420099 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:32.214121 6384 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.377882 6839 consensus_queue.cc:237] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.404915 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:32.408736 2476 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:32.443444 2481 raft_consensus.cc:2379] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:32.291972 6553 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.425746 2538 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:32.520402 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 183916us
I20260806 03:52:32.520578 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:32.372298 7007 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.525727 2574 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:32.527045 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:32.529402 2477 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:32.520010 7011 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:32.531934 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:32.532397 3584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 487) took 1534 ms. Trace:
W20260806 03:52:32.516245 3585 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.941s user 0.011s sys 0.002s
I20260806 03:52:32.552125 2584 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 5: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:32.413414 7014 raft_consensus.cc:2726] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:32.421658 6906 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.536015 6998 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:32.399322 6977 raft_consensus.cc:686] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [term 10 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:32.400225 6816 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:32.532124 6985 consensus_queue.cc:237] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:32.620050 6645 consensus_meta.cc:287] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.463s user 0.013s sys 0.004s
I20260806 03:52:32.406984 7005 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.399235 6851 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.533677 6836 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.402833 6855 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:32.372390 7009 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.297s
I20260806 03:52:32.395645 6770 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.616137 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:32.681172 2374 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 10: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:32.615195 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:32.682657 2370 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:52:32.621294 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:32.685052 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:32.630167 7028 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260806 03:52:32.631253 7031 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:32.541157 6899 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:32.688753 7028 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:32.689232 6899 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:32.689818 6899 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:32.689776 7028 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.640688 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:32.693595 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:32.695043 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:32.546322 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:32.696373 2373 raft_consensus.cc:3037] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:32.645154 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:32.699020 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:32.688709 7031 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.648311 6995 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:32.649205 7027 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:32.380362 7003 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.649817 6384 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:32.638659 6998 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:32.650918 7026 log.cc:924] Time spent T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260806 03:52:32.638969 6816 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:32.395141 6953 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.652917 6977 consensus_queue.cc:237] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.423769 5510 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.699994 2478 raft_consensus.cc:2379] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:32.703913 7027 raft_consensus.cc:686] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:32.707932 6816 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:32.712455 2577 raft_consensus.cc:2354] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 9. Current term is 10.
I20260806 03:52:32.713881 5510 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:32.719853 7027 consensus_queue.cc:237] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.674386 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:32.757134 2480 raft_consensus.cc:3037] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:32.624153 6925 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:32.760545 6925 raft_consensus.cc:686] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:32.761848 6925 consensus_queue.cc:237] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.737457 7028 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.745103 7003 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:32.657369 2476 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
I20260806 03:52:32.755786 7031 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:32.776966 7031 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.778764 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:32.750250 6998 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.777998 7031 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.796990 2574 raft_consensus.cc:2379] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:32.556058 3585 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 10.
I20260806 03:52:32.624836 7025 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:32.799726 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:32.676476 6991 raft_consensus.cc:979] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:52:32.833261 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:32.833452 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:32.839697 2481 raft_consensus.cc:2396] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260806 03:52:32.431314 7015 consensus_queue.cc:1046] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.421s
I20260806 03:52:32.669907 6861 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.676379 6855 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:32.680862 6836 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:32.847170 3547 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:32.542213 7007 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.548343 3584 rpcz_store.cc:276] 0806 03:52:30.998506 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.005650 (+ 7144us) service_pool.cc:227] Handling call
0806 03:52:31.044745 (+ 39095us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:32.248200 (+1203455us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:32.532119 (+283919us) spinlock_profiling.cc:243] Waited 993 ms on lock 0x7b540027a954. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:32.532325 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2283741056}
I20260806 03:52:32.828408 7012 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.677327 6851 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:32.858414 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:32.918839 2578 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:32.929066 2480 raft_consensus.cc:2454] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:32.541667 6996 consensus_queue.cc:237] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.961782 2480 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
W20260806 03:52:32.962213 2581 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s user 0.003s sys 0.044s
W20260806 03:52:32.971551 2581 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.805s user 0.003s sys 0.044s
I20260806 03:52:32.877455 2370 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:32.834836 7025 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:32.930083 2373 raft_consensus.cc:2454] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 3.
W20260806 03:52:32.941833 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 516208us
I20260806 03:52:32.837666 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:32.846743 6991 raft_consensus.cc:1067] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
W20260806 03:52:32.873157 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 506557us
I20260806 03:52:32.985751 2332 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:32.987171 2332 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:32.841907 3585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 478) took 1600 ms (client timeout 1913 ms). Trace:
I20260806 03:52:32.967764 2377 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260806 03:52:32.979243 2538 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:32.989518 2332 leader_election.cc:398] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6.
I20260806 03:52:32.989938 2332 leader_election.cc:401] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260806 03:52:32.992101 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:32.993281 2368 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:32.994647 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
W20260806 03:52:32.980566 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 275220us
I20260806 03:52:32.859165 7031 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.939057 6906 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:33.001716 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:33.014631 2368 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:32.961743 7006 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:32.962143 6894 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:33.030064 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:33.031164 2368 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:32.654356 6993 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:32.992264 6384 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:32.986079 6997 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.992604 6905 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:32.983229 7025 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:32.992035 7028 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:33.004647 6861 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:32.678371 7005 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:33.003793 6934 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:33.000051 3587 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:33.018203 6995 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:32.886600 6836 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:33.038916 6926 consensus_meta.cc:287] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.193s user 0.013s sys 0.002s
I20260806 03:52:32.991803 7003 consensus_queue.cc:1046] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260806 03:52:33.062933 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:32.993113 3585 rpcz_store.cc:269] 0806 03:52:31.234890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.253432 (+ 18542us) service_pool.cc:227] Handling call
0806 03:52:32.830517 (+1577085us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:33.045356 6553 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:33.045940 6907 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:33.077885 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:33.079381 2479 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 422) took 2874 ms (client timeout 2141 ms). Trace:
I20260806 03:52:33.085446 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:33.085983 3542 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:33.088920 2368 raft_consensus.cc:2396] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260806 03:52:33.086184 2479 rpcz_store.cc:269] 0806 03:52:30.194374 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.194675 (+ 301us) service_pool.cc:227] Handling call
0806 03:52:33.068087 (+2873412us) spinlock_profiling.cc:243] Waited 784 ms on lock 0x7b54000af0d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.068737 (+ 650us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1802890496}
I20260806 03:52:33.089814 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:33.090315 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:33.091150 2479 raft_consensus.cc:2379] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 5: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:33.093097 2537 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:33.094287 2327 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.734s user 0.008s sys 0.022s
I20260806 03:52:33.027383 6906 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:33.094697 2327 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.958s user 0.008s sys 0.023s
I20260806 03:52:33.095324 7035 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:33.095958 7035 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:33.096616 6906 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:33.089133 6926 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:33.098376 7035 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:33.099807 6926 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:33.057390 6861 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:33.100347 2574 raft_consensus.cc:1226] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260806 03:52:33.105577 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:33.090438 2368 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:33.117647 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:33.118654 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2302) took 1067 ms. Trace:
I20260806 03:52:33.110180 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:33.119192 2478 rpcz_store.cc:276] 0806 03:52:32.050922 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.781323 (+730401us) service_pool.cc:227] Handling call
0806 03:52:32.804667 (+ 23344us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:32.804945 (+ 278us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:52:32.806267 (+ 1322us) log.cc:842] Serialized 14 byte log entry
0806 03:52:33.118189 (+311922us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:33.118243 (+ 54us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:33.118289 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:33.118359 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:33.118482 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7119,"prepare.queue_time_us":348,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":96,"raft.queue_time_us":245445,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":7333,"thread_start_us":310141,"threads_started":1}
W20260806 03:52:33.099934 7046 log.cc:924] Time spent T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.116s user 0.001s sys 0.000s
I20260806 03:52:33.080236 6934 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:32.966156 7012 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:33.102837 7003 raft_consensus.cc:3032] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:33.091048 7031 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:33.087764 6997 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:33.101025 7032 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:33.037184 6894 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:33.056595 6384 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:33.114156 6770 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:33.129684 6906 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:33.273630 6904 consensus_meta.cc:287] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.221s user 0.013s sys 0.000s
I20260806 03:52:33.274240 6904 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:33.274920 2374 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:33.103446 6907 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:33.279286 6836 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:33.230283 2372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 2380) took 2395 ms (client timeout 1522 ms). Trace:
I20260806 03:52:33.114617 6861 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:33.281344 2372 rpcz_store.cc:269] 0806 03:52:30.834972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.842891 (+ 7919us) service_pool.cc:227] Handling call
0806 03:52:33.230029 (+2387138us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54003ea8d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.230152 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2711435648}
I20260806 03:52:33.233253 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2408) took 1173 ms. Trace:
I20260806 03:52:33.234202 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 2297) took 1215 ms. Trace:
I20260806 03:52:33.235145 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 440) took 1176 ms. Trace:
I20260806 03:52:33.144734 2376 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 LEADER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260806 03:52:33.128966 6698 consensus_meta.cc:287] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.002s user 0.011s sys 0.003s
I20260806 03:52:33.084064 7044 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:33.303373 2481 rpcz_store.cc:276] 0806 03:52:32.059008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.531605 (+472597us) service_pool.cc:227] Handling call
0806 03:52:33.234861 (+703256us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b540019fd54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.235052 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":683953024}
I20260806 03:52:33.333781 2374 raft_consensus.cc:2454] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
W20260806 03:52:33.553175 2534 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.018s sys 0.019s
I20260806 03:52:33.232230 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 439) took 1204 ms. Trace:
W20260806 03:52:33.800573 2534 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.810s user 0.019s sys 0.044s
I20260806 03:52:33.303140 2476 rpcz_store.cc:276] 0806 03:52:32.018668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.028511 (+ 9843us) service_pool.cc:227] Handling call
0806 03:52:33.233964 (+1205453us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54000af5d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.234121 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":494015744}
W20260806 03:52:33.801774 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 811260us
I20260806 03:52:33.304170 2577 rpcz_store.cc:276] 0806 03:52:32.060201 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.317048 (+256847us) service_pool.cc:227] Handling call
0806 03:52:33.232947 (+915899us) spinlock_profiling.cc:243] Waited 296 ms on lock 0x7b540024dbd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.233173 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":681929472}
I20260806 03:52:33.677767 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2407) took 1130 ms. Trace:
I20260806 03:52:33.803645 2379 rpcz_store.cc:276] 0806 03:52:32.547049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.686118 (+139069us) service_pool.cc:227] Handling call
0806 03:52:32.686408 (+ 290us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:33.125718 (+439310us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:33.125751 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:33.125800 (+ 49us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:33.125829 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:33.677289 (+551460us) spinlock_profiling.cc:243] Waited 439 ms on lock 0x7b54000c0654. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:33.677373 (+ 84us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:33.677645 (+ 272us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1009807744}
I20260806 03:52:33.805429 6698 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:33.515010 2332 leader_election.cc:334] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 1.693s (SENT)
I20260806 03:52:33.519850 6645 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:33.807104 2332 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:33.430294 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.038s user 0.002s sys 0.023s
W20260806 03:52:33.687208 2477 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2366) took 1646 ms (client timeout 1693 ms). Trace:
W20260806 03:52:33.658087 6855 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.795s user 0.012s sys 0.003s
W20260806 03:52:33.504840 2438 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.009s sys 0.015s
W20260806 03:52:33.808498 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 293573us
W20260806 03:52:33.469127 2578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1158) took 3071 ms (client timeout 1700 ms). Trace:
I20260806 03:52:33.303710 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:33.721593 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2295) took 1703 ms. Trace:
I20260806 03:52:33.800719 2484 rpcz_store.cc:276] 0806 03:52:32.028384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.028729 (+ 345us) service_pool.cc:227] Handling call
0806 03:52:32.029190 (+ 461us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:32.214203 (+185013us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:32.214235 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:32.214280 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:32.214312 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:33.231865 (+1017553us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54000afad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:33.231933 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:33.232144 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":424953216}
I20260806 03:52:33.143529 6553 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:52:33.811499 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.432s user 0.003s sys 0.023s
I20260806 03:52:33.811926 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:33.815137 2436 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:33.813129 2477 rpcz_store.cc:269] 0806 03:52:32.040308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.207139 (+166831us) service_pool.cc:227] Handling call
0806 03:52:33.686871 (+1479732us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b54002ffdd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.687098 (+ 227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":699561216}
I20260806 03:52:33.806026 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:33.198534 6961 consensus_meta.cc:287] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.386s user 0.011s sys 0.004s
I20260806 03:52:33.913620 2478 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
W20260806 03:52:33.869076 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 55120.6us
I20260806 03:52:33.231686 7048 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:33.176494 7027 raft_consensus.cc:2726] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689) with higher term. Message: Invalid argument: T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 5. Current term is 6.
W20260806 03:52:33.568642 6899 consensus_meta.cc:287] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.878s user 0.014s sys 0.004s
W20260806 03:52:33.442605 6920 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.894s user 0.005s sys 0.019s
I20260806 03:52:33.236469 7063 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:33.911401 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 99456.2us
I20260806 03:52:33.970854 2477 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb91ba465b9845129828f07c4d452970"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54220
W20260806 03:52:33.368734 6816 consensus_meta.cc:287] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.648s user 0.011s sys 0.010s
I20260806 03:52:33.310811 7032 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:33.535013 2576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1150) took 3165 ms (client timeout 1786 ms). Trace:
W20260806 03:52:33.284483 7011 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.603s user 0.000s sys 0.019s
W20260806 03:52:33.613289 6783 consensus_meta.cc:287] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.293s user 0.014s sys 0.003s
I20260806 03:52:33.807287 6645 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:33.916460 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:34.208056 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2405) took 1661 ms. Trace:
W20260806 03:52:33.791383 7028 consensus_meta.cc:287] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.729s user 0.013s sys 0.004s
W20260806 03:52:33.968713 2578 rpcz_store.cc:269] 0806 03:52:30.398195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.454723 (+ 56528us) service_pool.cc:227] Handling call
0806 03:52:33.468805 (+3014082us) spinlock_profiling.cc:243] Waited 2.21 s on lock 0x7b540013b0d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.468990 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5084543872}
W20260806 03:52:33.913893 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 98864.9us
I20260806 03:52:33.144099 7050 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:34.409471 2436 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:34.091676 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:33.927856 2538 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:34.410861 2477 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
W20260806 03:52:33.911731 2438 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.091s user 0.014s sys 0.020s
W20260806 03:52:34.409363 2577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2411) took 1410 ms (client timeout 1516 ms). Trace:
W20260806 03:52:33.721917 7025 consensus_meta.cc:287] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.658s user 0.008s sys 0.008s
W20260806 03:52:34.384776 7092 log.cc:924] Time spent T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.161s user 0.001s sys 0.000s
I20260806 03:52:34.421644 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:33.694562 7017 raft_consensus.cc:3032] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Stepping down as leader of term 6
W20260806 03:52:34.413368 2576 rpcz_store.cc:269] 0806 03:52:30.369830 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.592442 (+222612us) service_pool.cc:227] Handling call
0806 03:52:33.534732 (+2942290us) spinlock_profiling.cc:243] Waited 437 ms on lock 0x7b540040f1d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.534899 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1004807936}
I20260806 03:52:33.303020 6953 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:33.193303 7045 log.cc:924] Time spent T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
W20260806 03:52:34.417071 2577 rpcz_store.cc:269] 0806 03:52:32.999488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.803887 (+804399us) service_pool.cc:227] Handling call
0806 03:52:34.409265 (+605378us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:33.480891 6841 consensus_meta.cc:287] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.233s user 0.012s sys 0.003s
I20260806 03:52:33.519709 6765 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:33.566419 6907 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:34.418460 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 490694us
I20260806 03:52:33.929042 6983 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:33.283306 7005 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:33.232458 7047 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:33.570096 6972 log.cc:924] Time spent T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 2.244s user 0.005s sys 0.014s
I20260806 03:52:33.910940 7069 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:33.396772 6770 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 1.040s
I20260806 03:52:34.452322 7101 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.042774 7048 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.419842 7028 raft_consensus.cc:2726] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:34.454003 7101 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.454957 7028 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:34.455497 7028 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.456386 7101 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.456974 7101 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:34.457623 7028 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.452713 7069 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:34.458202 7028 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:34.455778 7048 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.435055 7017 raft_consensus.cc:729] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:34.458566 7069 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:33.200807 7007 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:33.921824 6698 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:33.301322 6851 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:34.459815 7017 consensus_queue.cc:260] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:33.147570 6934 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.448753 6765 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.462131 7069 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.462527 6934 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:33.815105 7004 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.457500 7101 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.463235 6765 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:33.224494 7012 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:33.568540 3587 raft_consensus.cc:1226] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260806 03:52:34.415729 6816 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.448436 6841 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.464910 7101 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.464915 6901 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.465742 6816 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:34.452044 7047 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:33.922971 6906 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.653s user 0.011s sys 0.004s
I20260806 03:52:33.924469 6904 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:33.995647 7083 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.466820 6906 raft_consensus.cc:2726] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:33.972078 6926 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.466948 6904 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.467631 6926 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:34.468899 6861 consensus_meta.cc:287] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 1.086s user 0.017s sys 0.000s
I20260806 03:52:34.468981 7047 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.469367 6904 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.469805 6926 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:34.469446 6861 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:33.929273 6839 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:33.912758 6553 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:33.911247 6384 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:34.414592 6920 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.878s user 0.016s sys 0.029s
I20260806 03:52:33.995012 6855 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.419602 6899 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:33.999680 6918 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:34.419745 7050 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:34.061702 7027 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:34.424356 6645 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:34.424993 6857 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.423889 7025 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.458757 7028 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.422860 6862 raft_consensus.cc:2781] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:34.392630 6961 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:34.406381 7011 raft_consensus.cc:686] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:34.452723 7032 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:33.174556 7003 raft_consensus.cc:729] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:33.365546 6991 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.426463 7063 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.485695 6991 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.485911 7011 consensus_queue.cc:237] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.486032 7003 consensus_queue.cc:260] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.487221 7063 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:33.307843 6997 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:33.307411 7031 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.487974 6991 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:34.478029 7027 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.489346 6997 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.489529 7031 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.475787 7017 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:34.450326 6983 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.467099 7083 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.490509 7027 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.491851 7083 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.487515 7003 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:34.476464 7050 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:33.910764 7044 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.496847 7028 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.496965 7044 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.494244 7025 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:34.544203 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.002s sys 0.006s
I20260806 03:52:34.545133 2373 raft_consensus.cc:1226] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9. Ops: []
W20260806 03:52:34.545944 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.575s user 0.000s sys 0.003s
W20260806 03:52:34.546578 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.576s user 0.000s sys 0.003s
I20260806 03:52:34.389614 2376 rpcz_store.cc:276] 0806 03:52:32.546195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.548561 (+ 2366us) service_pool.cc:227] Handling call
0806 03:52:32.548969 (+ 408us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:33.328286 (+779317us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.207684 (+879398us) spinlock_profiling.cc:243] Waited 595 ms on lock 0x7b540026f054. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.207944 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1369220480}
W20260806 03:52:34.548404 2372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 480) took 1430 ms (client timeout 1665 ms). Trace:
W20260806 03:52:34.549156 2372 rpcz_store.cc:269] 0806 03:52:33.117664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.302361 (+184697us) service_pool.cc:227] Handling call
0806 03:52:34.547977 (+1245616us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.550102 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.552155 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1257) took 1070 ms. Trace:
I20260806 03:52:34.545603 2368 raft_consensus.cc:1226] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:34.552744 2376 rpcz_store.cc:276] 0806 03:52:33.481423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.548962 (+1067539us) service_pool.cc:227] Handling call
0806 03:52:34.552054 (+ 3092us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:34.553438 2375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 463) took 3742 ms (client timeout 1971 ms). Trace:
W20260806 03:52:34.552772 2327 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.004s sys 0.012s
W20260806 03:52:34.554236 2327 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.541s user 0.004s sys 0.013s
I20260806 03:52:34.554129 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:34.554651 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2420) took 1563 ms. Trace:
W20260806 03:52:34.555364 2369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1252) took 3466 ms (client timeout 3359 ms). Trace:
I20260806 03:52:34.555192 2368 rpcz_store.cc:276] 0806 03:52:32.991214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.114272 (+123058us) service_pool.cc:227] Handling call
0806 03:52:33.114607 (+ 335us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.553365 (+1438758us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.554281 (+ 916us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54001eafd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.554552 (+ 271us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3289329408}
I20260806 03:52:34.556648 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2406) took 2010 ms. Trace:
W20260806 03:52:34.557008 2374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 2411) took 1917 ms (client timeout 1887 ms). Trace:
W20260806 03:52:34.545593 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.002s sys 0.006s
I20260806 03:52:34.557145 2377 rpcz_store.cc:276] 0806 03:52:32.546704 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.633031 (+ 86327us) service_pool.cc:227] Handling call
0806 03:52:32.647167 (+ 14136us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:32.988620 (+341453us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.556055 (+1567435us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b540005a5d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.556453 (+ 398us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":736439040}
W20260806 03:52:34.557730 2374 rpcz_store.cc:269] 0806 03:52:32.639020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.682242 (+ 43222us) service_pool.cc:227] Handling call
0806 03:52:34.551694 (+1869452us) spinlock_profiling.cc:243] Waited 440 ms on lock 0x7b54004295d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:34.552500 (+ 806us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1012044160}
I20260806 03:52:34.559370 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.559950 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.560411 732 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260806 03:52:34.556286 2369 rpcz_store.cc:269] 0806 03:52:31.088826 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.178934 (+ 90108us) service_pool.cc:227] Handling call
0806 03:52:34.554672 (+3375738us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5400331dd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:34.555019 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":315472768}
I20260806 03:52:34.561362 2374 raft_consensus.cc:2454] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:34.562309 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.562700 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1259) took 1080 ms. Trace:
I20260806 03:52:34.563508 2369 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 9.
W20260806 03:52:34.554032 2375 rpcz_store.cc:269] 0806 03:52:30.810705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.815804 (+ 5099us) service_pool.cc:227] Handling call
0806 03:52:34.552778 (+3736974us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b54003331d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:34.553091 (+ 313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2268903296}
I20260806 03:52:34.546751 2370 raft_consensus.cc:1226] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:34.565491 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2416) took 1575 ms. Trace:
I20260806 03:52:34.565608 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.566720 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2415) took 1577 ms. Trace:
I20260806 03:52:34.566093 2373 rpcz_store.cc:276] 0806 03:52:32.989832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.008431 (+ 18599us) service_pool.cc:227] Handling call
0806 03:52:33.008741 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.546428 (+1537687us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.565073 (+ 18645us) spinlock_profiling.cc:243] Waited 1.54 s on lock 0x7b54000596d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.565391 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3531045760}
I20260806 03:52:34.566692 2375 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:34.555375 2376 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:34.567804 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1264) took 1084 ms. Trace:
I20260806 03:52:34.568533 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.559863 2331 leader_election.cc:398] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 9. Current term is 10.
I20260806 03:52:34.569762 2331 leader_election.cc:401] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Cancelling election due to peer responding with higher term
I20260806 03:52:34.569994 2373 raft_consensus.cc:2379] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 7: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:34.570667 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1258) took 1089 ms. Trace:
I20260806 03:52:34.571079 2373 rpcz_store.cc:276] 0806 03:52:33.481756 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.567937 (+1086181us) service_pool.cc:227] Handling call
0806 03:52:34.570593 (+ 2656us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.560577 2377 raft_consensus.cc:2379] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 7: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:34.568238 2375 rpcz_store.cc:276] 0806 03:52:33.483699 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.565118 (+1081419us) service_pool.cc:227] Handling call
0806 03:52:34.567722 (+ 2604us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.572765 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1254) took 1092 ms. Trace:
I20260806 03:52:34.567293 2370 rpcz_store.cc:276] 0806 03:52:32.989407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.995684 (+ 6277us) service_pool.cc:227] Handling call
0806 03:52:32.996148 (+ 464us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.565657 (+1569509us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.566364 (+ 707us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54001072d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.566652 (+ 288us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3564350720}
I20260806 03:52:34.573279 2377 rpcz_store.cc:276] 0806 03:52:33.480487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.558845 (+1078358us) service_pool.cc:227] Handling call
0806 03:52:34.572663 (+ 13818us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.574868 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2422) took 1567 ms. Trace:
I20260806 03:52:34.575167 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 478) took 1566 ms. Trace:
I20260806 03:52:34.575549 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 481) took 1457 ms. Trace:
I20260806 03:52:34.563798 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:52:34.563779 2374 rpcz_store.cc:276] 0806 03:52:33.482084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.559422 (+1077338us) service_pool.cc:227] Handling call
0806 03:52:34.562516 (+ 3094us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.557238 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.551637 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.577651 2379 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:34.578375 2372 raft_consensus.cc:2379] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:34.576658 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:34.580622 2373 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:34.581496 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 479) took 1464 ms. Trace:
I20260806 03:52:34.582017 2373 rpcz_store.cc:276] 0806 03:52:33.117311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.571974 (+1454663us) service_pool.cc:227] Handling call
0806 03:52:34.581382 (+ 9408us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.575647 2370 rpcz_store.cc:276] 0806 03:52:33.009085 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.573688 (+1564603us) service_pool.cc:227] Handling call
0806 03:52:34.574315 (+ 627us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.574756 (+ 441us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:34.574790 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:34.574879 (+ 89us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:34.574923 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.574990 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.575103 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.575395 2375 rpcz_store.cc:276] 0806 03:52:33.007619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.573017 (+1565398us) service_pool.cc:227] Handling call
0806 03:52:34.573730 (+ 713us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.574272 (+ 542us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:34.574326 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:34.574403 (+ 77us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:34.574512 (+ 109us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.574597 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.574752 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.586647 2370 raft_consensus.cc:1261] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 03:52:34.586714 2375 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:34.587292 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1255) took 1106 ms. Trace:
I20260806 03:52:34.564141 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1256) took 1083 ms. Trace:
I20260806 03:52:34.576074 2377 rpcz_store.cc:276] 0806 03:52:33.117972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.574225 (+1456253us) service_pool.cc:227] Handling call
0806 03:52:34.574813 (+ 588us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.575205 (+ 392us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:34.575235 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:34.575292 (+ 57us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:34.575323 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.575386 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.575499 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.589004 2369 rpcz_store.cc:276] 0806 03:52:33.481089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.561780 (+1080691us) service_pool.cc:227] Handling call
0806 03:52:34.564081 (+ 2301us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.578931 2368 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:34.590727 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1262) took 1107 ms. Trace:
W20260806 03:52:34.592129 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 589424us
I20260806 03:52:34.416920 2477 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Advancing to term 10
W20260806 03:52:34.592880 2379 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37370 (request call id 483) took 1290 ms (client timeout 1642 ms). Trace:
I20260806 03:52:34.591126 2368 rpcz_store.cc:276] 0806 03:52:33.483062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.556791 (+1073729us) service_pool.cc:227] Handling call
0806 03:52:34.590588 (+ 33797us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.416805 2478 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:34.591187 2369 raft_consensus.cc:1261] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260806 03:52:33.998868 2482 rpcz_store.cc:276] 0806 03:52:32.017997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.018424 (+ 427us) service_pool.cc:227] Handling call
0806 03:52:32.019372 (+ 948us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:32.057846 (+ 38474us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:32.057898 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:32.085584 (+ 27686us) log.cc:842] Serialized 26 byte log entry
0806 03:52:32.236896 (+151312us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:32.253258 (+ 16362us) log.cc:842] Serialized 14 byte log entry
0806 03:52:32.253508 (+ 250us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:32.253567 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:33.679699 (+1426132us) spinlock_profiling.cc:243] Waited 34.9 ms on lock 0x7b18003e1d00. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:33.710837 (+ 31138us) raft_consensus.cc:1671] finished
0806 03:52:33.721190 (+ 10353us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:33.721481 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13742,"prepare.queue_time_us":19431,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":61,"raft.queue_time_us":122249,"raft.run_cpu_time_us":593,"raft.run_wall_time_us":82514,"spinlock_wait_cycles":80243328,"thread_start_us":63991,"threads_started":1,"wal-append.queue_time_us":206827}
I20260806 03:52:33.970993 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:34.579881 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1263) took 1096 ms. Trace:
I20260806 03:52:34.441001 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:34.609822 2372 rpcz_store.cc:276] 0806 03:52:33.483382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.551100 (+1067718us) service_pool.cc:227] Handling call
0806 03:52:34.579787 (+ 28687us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:34.611073 2576 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 381) took 1529 ms (client timeout 1718 ms). Trace:
W20260806 03:52:34.611263 2572 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1147) took 4291 ms (client timeout 1597 ms). Trace:
W20260806 03:52:34.611680 2576 rpcz_store.cc:269] 0806 03:52:33.081189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.440691 (+1359502us) service_pool.cc:227] Handling call
0806 03:52:34.610767 (+170076us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:34.611948 2572 rpcz_store.cc:269] 0806 03:52:30.320035 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.349556 (+ 29521us) service_pool.cc:227] Handling call
0806 03:52:34.610611 (+4261055us) spinlock_profiling.cc:243] Waited 1.64 s on lock 0x7b54001897d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:34.610930 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3762017664}
I20260806 03:52:34.614012 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 448) took 1229 ms. Trace:
I20260806 03:52:34.615458 2575 raft_consensus.cc:1261] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:52:34.618861 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1169) took 4134 ms. Trace:
I20260806 03:52:34.619228 2538 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:34.434795 2534 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:34.619400 2575 rpcz_store.cc:276] 0806 03:52:30.484834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.217054 (+732220us) service_pool.cc:227] Handling call
0806 03:52:33.535207 (+2318153us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.616212 (+1081005us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.617223 (+ 1011us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540040f1d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.618399 (+ 1176us) spinlock_profiling.cc:243] Waited 2.32 s on lock 0x7b540040f1d0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:34.618717 (+ 318us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7812291712}
I20260806 03:52:34.439708 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:34.620960 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:34.621317 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1174) took 2558 ms. Trace:
I20260806 03:52:34.614622 2479 rpcz_store.cc:276] 0806 03:52:33.384409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.384732 (+ 323us) service_pool.cc:227] Handling call
0806 03:52:33.385276 (+ 544us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.598868 (+1213592us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:34.598950 (+ 82us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:34.599044 (+ 94us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:34.599082 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.613381 (+ 14299us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54002a12d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.613448 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.613685 (+ 237us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2789251328}
I20260806 03:52:34.622262 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2311) took 1544 ms. Trace:
I20260806 03:52:34.623696 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:34.614032 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:34.426087 2578 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 383) took 1295 ms (client timeout 1665 ms). Trace:
W20260806 03:52:34.593617 2379 rpcz_store.cc:269] 0806 03:52:33.302533 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.805647 (+503114us) service_pool.cc:227] Handling call
0806 03:52:34.592530 (+786883us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b540014e454. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:34.592786 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1396450560}
I20260806 03:52:34.585003 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 484) took 1119 ms. Trace:
I20260806 03:52:34.660858 2572 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 382) took 1530 ms. Trace:
I20260806 03:52:34.661527 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:34.631418 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2395) took 4461 ms. Trace:
W20260806 03:52:34.662155 2577 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2412) took 1351 ms (client timeout 1640 ms). Trace:
I20260806 03:52:34.662747 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 379) took 1675 ms. Trace:
W20260806 03:52:34.662701 2577 rpcz_store.cc:269] 0806 03:52:33.310843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.439331 (+1128488us) service_pool.cc:227] Handling call
0806 03:52:34.661933 (+222602us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.663542 2481 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:34.661582 2572 rpcz_store.cc:276] 0806 03:52:33.130372 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.613344 (+1482972us) service_pool.cc:227] Handling call
0806 03:52:34.660699 (+ 47355us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.665128 2476 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:34.665942 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54220 (request call id 2312) took 1042 ms. Trace:
I20260806 03:52:34.662531 2579 rpcz_store.cc:276] 0806 03:52:30.169866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.181912 (+1012046us) service_pool.cc:227] Handling call
0806 03:52:31.182392 (+ 480us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:32.196534 (+1014142us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.630616 (+2434082us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b540028ebd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.631158 (+ 542us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1879847808}
I20260806 03:52:34.666442 2476 rpcz_store.cc:276] 0806 03:52:33.623426 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.801951 (+178525us) service_pool.cc:227] Handling call
0806 03:52:34.596985 (+795034us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:52:34.665866 (+ 68881us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23267968}
I20260806 03:52:34.664453 2481 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56244 (request call id 449) took 1222 ms. Trace:
I20260806 03:52:34.669157 2368 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:52:34.669253 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 375) took 2611 ms. Trace:
I20260806 03:52:34.665814 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:34.103044 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:34.470620 6553 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:33.534049 3588 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 476) took 2724 ms (client timeout 1872 ms). Trace:
W20260806 03:52:34.686718 2583 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 372) took 2631 ms (client timeout 2716 ms). Trace:
W20260806 03:52:34.687464 3588 rpcz_store.cc:269] 0806 03:52:30.809262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:30.809572 (+ 310us) service_pool.cc:227] Handling call
0806 03:52:33.533449 (+2723877us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b540052f654. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:33.533826 (+ 377us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":569927424}
I20260806 03:52:34.675410 6799 consensus_queue.cc:1035] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260806 03:52:34.689422 2584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56018 (request call id 371) took 3371 ms (client timeout 1718 ms). Trace:
I20260806 03:52:34.689622 6861 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:34.689949 2584 rpcz_store.cc:269] 0806 03:52:31.317536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.317700 (+ 164us) service_pool.cc:227] Handling call
0806 03:52:34.688797 (+3371097us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b540040f6d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:34.689143 (+ 346us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2715878784}
I20260806 03:52:33.911219 6785 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.681921 2481 rpcz_store.cc:276] 0806 03:52:33.441903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.797981 (+356078us) service_pool.cc:227] Handling call
0806 03:52:34.612558 (+814577us) spinlock_profiling.cc:243] Waited 873 us on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:52:34.623203 (+ 10645us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65b7c 00007f0d79b6573b 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a071b0 00007f0d81a06a94 00007f0d7b0eec04 00007f0d7b0eeb92
0806 03:52:34.664371 (+ 41168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22585472}
I20260806 03:52:34.676646 2377 raft_consensus.cc:1261] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:52:34.664281 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:34.465848 6841 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.465656 6901 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:34.661163 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 226300us
I20260806 03:52:34.665787 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:34.694380 6855 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.695125 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2414) took 1383 ms. Trace:
I20260806 03:52:34.695518 6809 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.695524 2576 rpcz_store.cc:276] 0806 03:52:33.311622 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.613275 (+1301653us) service_pool.cc:227] Handling call
0806 03:52:34.695014 (+ 81739us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":42880}
I20260806 03:52:34.696563 6809 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.697912 2576 raft_consensus.cc:1261] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 03:52:34.698582 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 376) took 2571 ms. Trace:
I20260806 03:52:34.698940 2576 rpcz_store.cc:276] 0806 03:52:32.127697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.696747 (+2569050us) service_pool.cc:227] Handling call
0806 03:52:34.697257 (+ 510us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.698283 (+ 1026us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.698501 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.699458 6809 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.700700 6901 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.470620 6839 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.680719 6926 raft_consensus.cc:2781] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Leader election won for term 3
I20260806 03:52:34.704717 2576 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 9. Current term is 10. Ops: []
I20260806 03:52:34.705155 6926 raft_consensus.cc:686] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [term 3 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:34.705495 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1176) took 2334 ms. Trace:
I20260806 03:52:34.705957 2576 rpcz_store.cc:276] 0806 03:52:32.371023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.699697 (+2328674us) service_pool.cc:227] Handling call
0806 03:52:34.704200 (+ 4503us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.705212 (+ 1012us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.705405 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.706606 6839 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.679483 6816 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:34.680193 6862 raft_consensus.cc:686] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:34.587695 2370 rpcz_store.cc:276] 0806 03:52:33.480807 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.585678 (+1104871us) service_pool.cc:227] Handling call
0806 03:52:34.586178 (+ 500us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.587040 (+ 862us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.587233 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.678144 2574 rpcz_store.cc:276] 0806 03:52:32.063091 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.970132 (+907041us) service_pool.cc:227] Handling call
0806 03:52:33.017429 (+ 47297us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:33.118749 (+101320us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.620559 (+1501810us) spinlock_profiling.cc:243] Waited 82.6 ms on lock 0x7b540004ae54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.621051 (+ 492us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":189899008}
I20260806 03:52:34.480369 6857 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.684691 6899 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.676627 6770 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:34.681955 2581 rpcz_store.cc:276] 0806 03:52:32.057866 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.057982 (+ 116us) service_pool.cc:227] Handling call
0806 03:52:32.058327 (+ 345us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:32.058722 (+ 395us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:32.058749 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:32.084689 (+ 25940us) log.cc:842] Serialized 52 byte log entry
0806 03:52:32.972167 (+887478us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:33.022423 (+ 50256us) log.cc:842] Serialized 14 byte log entry
0806 03:52:33.442688 (+420265us) log.cc:842] Serialized 14 byte log entry
0806 03:52:33.443632 (+ 944us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:33.443747 (+ 115us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:34.668577 (+1224830us) raft_consensus.cc:1671] finished
0806 03:52:34.668631 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.669021 (+ 390us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36977,"prepare.queue_time_us":40444,"prepare.run_cpu_time_us":163,"prepare.run_wall_time_us":7910,"raft.queue_time_us":79585,"raft.run_cpu_time_us":424,"raft.run_wall_time_us":30458,"thread_start_us":1188960,"threads_started":2,"wal-append.queue_time_us":910117}
I20260806 03:52:34.721217 6770 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.722505 2581 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 03:52:34.691664 6785 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.724607 6770 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.667976 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:34.701822 6918 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260806 03:52:34.727427 2579 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:34.701785 6907 consensus_queue.cc:1035] T f3ae6cf9eba9410d8307ed1da6a62ea6 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:34.730144 6857 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.732259 6841 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.732578 7025 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:34.733072 6841 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.735738 6899 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.736168 7050 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.736584 6899 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.664968 2373 rpcz_store.cc:276] 0806 03:52:33.465315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.583140 (+1117825us) service_pool.cc:227] Handling call
0806 03:52:34.583958 (+ 818us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.584532 (+ 574us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:34.584571 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:34.584663 (+ 92us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:34.584706 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.584792 (+ 86us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.584931 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.706805 6926 consensus_queue.cc:237] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.664678 2480 rpcz_store.cc:276] 0806 03:52:33.077934 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.113218 (+ 35284us) service_pool.cc:227] Handling call
0806 03:52:33.113624 (+ 406us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.604734 (+1491110us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:34.604810 (+ 76us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:34.604925 (+ 115us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:34.604980 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.621823 (+ 16843us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54002a17d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:34.621879 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.622073 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3427729280}
I20260806 03:52:34.663396 2575 rpcz_store.cc:276] 0806 03:52:32.987440 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.661023 (+1673583us) service_pool.cc:227] Handling call
0806 03:52:34.662656 (+ 1633us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:34.687378 2583 rpcz_store.cc:269] 0806 03:52:32.055304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.055485 (+ 181us) service_pool.cc:227] Handling call
0806 03:52:34.686235 (+2630750us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b5400155c54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:34.686473 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25326976}
I20260806 03:52:34.628631 2374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 482) took 1439 ms. Trace:
I20260806 03:52:34.718010 2577 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 380) took 1701 ms. Trace:
I20260806 03:52:34.707913 2576 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 03:52:34.742098 2577 rpcz_store.cc:276] 0806 03:52:33.009946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.702673 (+1692727us) service_pool.cc:227] Handling call
0806 03:52:34.703465 (+ 792us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.703902 (+ 437us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:34.703944 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:34.710785 (+ 6841us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:34.710857 (+ 72us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.710949 (+ 92us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.711091 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.743484 2374 rpcz_store.cc:276] 0806 03:52:33.189485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.577355 (+1387870us) service_pool.cc:227] Handling call
0806 03:52:34.592361 (+ 15006us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.628213 (+ 35852us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:34.628250 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:34.628297 (+ 47us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:34.628329 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.628402 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:34.628540 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:34.747417 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 153069us
I20260806 03:52:34.707561 6553 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.749562 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2413) took 1432 ms. Trace:
I20260806 03:52:34.753273 2576 rpcz_store.cc:276] 0806 03:52:33.311285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.706845 (+1395560us) service_pool.cc:227] Handling call
0806 03:52:34.707349 (+ 504us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.742964 (+ 35615us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.743205 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:34.757521 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 172076us
W20260806 03:52:34.778200 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 368639us
I20260806 03:52:34.793488 7063 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.794709 7063 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:34.810597 6983 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.810266 2477 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.812958 2375 raft_consensus.cc:1261] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:52:34.738860 6855 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:34.845054 2375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1260) took 1331 ms. Trace:
I20260806 03:52:34.845544 2375 rpcz_store.cc:276] 0806 03:52:33.482393 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.585940 (+1103547us) service_pool.cc:227] Handling call
0806 03:52:34.586515 (+ 575us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.813677 (+227162us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:34.813974 (+ 297us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.738731 6862 consensus_queue.cc:237] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:34.871017 2484 raft_consensus.cc:1226] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 9. Current term is 10. Ops: []
W20260806 03:52:34.857607 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 77456.7us
I20260806 03:52:34.882344 7072 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:34.855090 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 154469us
I20260806 03:52:34.874902 6785 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.873291 6553 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:34.857163 7050 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.880575 6899 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:34.885918 7005 consensus_queue.cc:1035] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:34.883332 6770 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:34.847065 2584 raft_consensus.cc:1261] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260806 03:52:34.857025 2477 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:34.883944 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 104582us
W20260806 03:52:34.906160 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 160363us
W20260806 03:52:34.877044 2578 rpcz_store.cc:269] 0806 03:52:33.130813 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.420892 (+1290079us) service_pool.cc:227] Handling call
0806 03:52:34.426014 (+ 5122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:34.887696 6894 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:35.080714 6987 log.cc:924] Time spent T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 3.219s user 0.017s sys 0.002s
W20260806 03:52:34.919970 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 164894us
I20260806 03:52:35.084272 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2280) took 3863 ms. Trace:
I20260806 03:52:35.075798 3590 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 493) took 2106 ms. Trace:
I20260806 03:52:34.936376 2376 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:35.101639 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1175) took 3032 ms. Trace:
I20260806 03:52:35.076618 2436 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:35.098102 2376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40734 (request call id 1253) took 1618 ms (client timeout 1690 ms). Trace:
W20260806 03:52:35.105978 2376 rpcz_store.cc:269] 0806 03:52:33.480088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.553625 (+1073537us) service_pool.cc:227] Handling call
0806 03:52:35.098004 (+544379us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:35.102231 2584 rpcz_store.cc:276] 0806 03:52:32.063348 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.691218 (+2627870us) service_pool.cc:227] Handling call
0806 03:52:34.691922 (+ 704us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:35.093177 (+401255us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:35.096119 (+ 2942us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:34.919809 6989 log.cc:924] Time spent T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 2.999s user 0.011s sys 0.011s
I20260806 03:52:34.987244 7063 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.047317 6861 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.008476 7113 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:35.094722 3590 rpcz_store.cc:276] 0806 03:52:32.968924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.969111 (+ 187us) service_pool.cc:227] Handling call
0806 03:52:33.008058 (+ 38947us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:33.116450 (+108392us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:33.116500 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:33.117120 (+ 620us) log.cc:842] Serialized 26 byte log entry
0806 03:52:33.628418 (+511298us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:33.628486 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:35.074735 (+1446249us) spinlock_profiling.cc:243] Waited 89.1 ms on lock 0x7b54004a2254. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:35.074818 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:35.074998 (+ 180us) raft_consensus.cc:1671] finished
0806 03:52:35.075013 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:35.075513 (+ 500us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18774,"raft.queue_time_us":27183,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":9148,"spinlock_wait_cycles":204895488,"thread_start_us":434515,"threads_started":1,"wal-append.queue_time_us":1244570}
I20260806 03:52:35.131815 7113 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.039971 6855 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.084759 2474 rpcz_store.cc:276] 0806 03:52:31.221050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.221238 (+ 188us) service_pool.cc:227] Handling call
0806 03:52:31.221580 (+ 342us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:31.221989 (+ 409us) raft_consensus.cc:1520] Early marking committed up to index 3
0806 03:52:31.348280 (+126291us) log.cc:842] Serialized 14 byte log entry
0806 03:52:31.551498 (+203218us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:31.559721 (+ 8223us) log.cc:842] Serialized 26 byte log entry
0806 03:52:31.559887 (+ 166us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:31.574334 (+ 14447us) log.cc:842] Serialized 14 byte log entry
0806 03:52:31.574737 (+ 403us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:31.574845 (+ 108us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:33.194923 (+1620078us) spinlock_profiling.cc:243] Waited 30.3 ms on lock 0x7b18003e1b20. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b6af96 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11
0806 03:52:33.224444 (+ 29521us) spinlock_profiling.cc:243] Waited 13.8 ms on lock 0x7b18003e1b20. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:34.440473 (+1216029us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b18003e1b20. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d764 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:35.039846 (+599373us) spinlock_profiling.cc:243] Waited 7.11 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:35.083701 (+ 43855us) raft_consensus.cc:1671] finished
0806 03:52:35.083729 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:35.083949 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":295752,"prepare.queue_time_us":388659,"prepare.run_cpu_time_us":132,"prepare.run_wall_time_us":4028,"raft.queue_time_us":24475,"raft.run_cpu_time_us":371,"raft.run_wall_time_us":5361,"spinlock_wait_cycles":141980928,"thread_start_us":298281,"threads_started":2,"wal-append.queue_time_us":285979}
I20260806 03:52:35.125485 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:54220 (request call id 2313) took 1293 ms. Trace:
I20260806 03:52:35.126573 2534 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:35.177723 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.174674 2477 rpcz_store.cc:276] 0806 03:52:33.816152 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:33.970723 (+154571us) service_pool.cc:227] Handling call
0806 03:52:35.103439 (+1132716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:35.175707 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:35.185137 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 72746.1us
I20260806 03:52:35.183188 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2415) took 1337 ms. Trace:
W20260806 03:52:35.184401 3545 leader_election.cc:334] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407): Timed out: RequestConsensusVote RPC to 127.0.183.3:46407 timed out after 1.718s (SENT)
I20260806 03:52:35.185837 2574 rpcz_store.cc:276] 0806 03:52:33.846159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.739493 (+893334us) service_pool.cc:227] Handling call
0806 03:52:34.778475 (+ 38982us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.779212 (+ 737us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 03:52:34.987151 (+207939us) log.cc:842] Serialized 14 byte log entry
0806 03:52:35.182732 (+195581us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:35.182798 (+ 66us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:35.182848 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:35.182943 (+ 95us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:35.183113 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":72232,"prepare.queue_time_us":339298,"raft.queue_time_us":86922,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":207,"thread_start_us":236229,"threads_started":2}
W20260806 03:52:35.198954 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 85958.7us
I20260806 03:52:35.193552 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:35.195411 6861 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.210162 6993 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 03:52:35.177574 6770 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.205811 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.177505 7072 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.211053 6861 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.206411 7025 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: Vote denied by peer fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407) with higher term. Message: Invalid argument: T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for earlier term 9. Current term is 10.
I20260806 03:52:35.179564 6915 consensus_queue.cc:1046] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260806 03:52:35.222723 7063 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.225957 2534 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:35.228750 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2291) took 3974 ms. Trace:
W20260806 03:52:35.235304 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 158881us
W20260806 03:52:35.231587 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 115437us
W20260806 03:52:35.238415 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 62443.7us
I20260806 03:52:35.231494 2483 rpcz_store.cc:276] 0806 03:52:31.254110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:31.318543 (+ 64433us) service_pool.cc:227] Handling call
0806 03:52:31.319044 (+ 501us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:31.326334 (+ 7290us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:31.326368 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:31.336490 (+ 10122us) log.cc:842] Serialized 26 byte log entry
0806 03:52:31.546131 (+209641us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:31.546174 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:31.546250 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:34.912338 (+3366088us) spinlock_profiling.cc:243] Waited 6.69 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:35.228093 (+315755us) raft_consensus.cc:1671] finished
0806 03:52:35.228122 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:35.228550 (+ 428us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7003,"raft.queue_time_us":7473,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":30048,"spinlock_wait_cycles":15392512,"thread_start_us":139560,"threads_started":1,"wal-append.queue_time_us":458299}
I20260806 03:52:35.247171 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.252216 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.253580 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.259764 2376 raft_consensus.cc:2396] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260806 03:52:35.260196 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:35.253301 7063 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.229043 7111 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260806 03:52:35.265590 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.270744 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.266557 2534 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:35.275235 2574 raft_consensus.cc:2379] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 4: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:35.279407 2584 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:35.282423 2376 raft_consensus.cc:1226] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:35.277558 2538 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:35.284335 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.270478 6926 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:35.288280 2375 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:35.286352 6915 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:35.279886 7025 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.289640 7072 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.286247 6926 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:35.302873 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.291730 6861 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.305218 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.307050 7063 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.295122 6855 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.294612 7113 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.307863 6770 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.301139 6926 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.312507 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.307063 7068 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:35.314541 7068 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:35.315765 7068 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.318747 7025 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.320600 6489 raft_consensus.cc:3032] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:35.321043 6489 raft_consensus.cc:729] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:35.321337 7025 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.323518 6861 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.323770 6489 consensus_queue.cc:260] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.325426 2375 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:35.326527 6489 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:35.329324 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:35.333223 2429 connection.cc:232] RPC call timeout handler was delayed by -1.5290876748040318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20260806 03:52:35.333936 2429 connection.cc:232] RPC call timeout handler was delayed by -1.4917341119144112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1627s for a response
W20260806 03:52:35.334285 2429 connection.cc:232] RPC call timeout handler was delayed by -1.2600711639970541s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1899s for a response
W20260806 03:52:35.334621 2429 connection.cc:232] RPC call timeout handler was delayed by -1.1749927077908069s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33590000000000003s for a response
I20260806 03:52:35.313514 3542 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:35.337474 2584 raft_consensus.cc:2379] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:35.344398 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:35.346341 2429 leader_election.cc:334] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.627s (SENT)
I20260806 03:52:35.347186 2429 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.349622 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:35.350358 2584 raft_consensus.cc:2362] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7. Re-sending same reply.
I20260806 03:52:35.350492 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.351312 2576 raft_consensus.cc:2379] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:35.352188 2584 raft_consensus.cc:1261] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:35.352324 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.352555 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.353618 2572 raft_consensus.cc:2379] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 9: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:35.355078 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.355070 6815 consensus_queue.cc:1046] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:35.355765 2534 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:35.356161 2579 raft_consensus.cc:2454] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 9.
W20260806 03:52:35.356364 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 50933.8us
I20260806 03:52:35.356601 6815 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:52:35.357901 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.360448 6906 consensus_queue.cc:1035] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260806 03:52:35.361234 7015 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:35.363795 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.364082 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.364444 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.364760 2373 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:35.365644 2368 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 10: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:35.365978 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: true dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:35.366580 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.367094 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:35.367813 2370 raft_consensus.cc:2362] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8. Re-sending same reply.
I20260806 03:52:35.368863 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.369473 2370 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:35.371043 6770 raft_consensus.cc:2726] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:35.357162 7050 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:35.353497 2578 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:35.370385 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.374795 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.375474 2370 raft_consensus.cc:2454] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 2.
I20260806 03:52:35.373440 2581 raft_consensus.cc:1226] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:35.372252 7050 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:35.377537 7050 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:35.366603 2373 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:35.382675 2369 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 03:52:35.367879 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.364627 2379 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:35.388640 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.390693 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40734 (request call id 1261) took 1900 ms. Trace:
I20260806 03:52:35.391252 2377 rpcz_store.cc:276] 0806 03:52:33.482705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.589836 (+1107131us) service_pool.cc:227] Handling call
0806 03:52:34.676169 (+ 86333us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:34.711209 (+ 35040us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:35.376834 (+665625us) spinlock_profiling.cc:243] Waited 85.8 ms on lock 0x7b540014e450. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:35.383241 (+ 6407us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":197375360}
I20260806 03:52:35.394914 2374 raft_consensus.cc:2379] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 7: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:35.403199 2376 raft_consensus.cc:2454] T d7a939d2320d4a478b95733e1579e2b3 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:35.411199 6815 raft_consensus.cc:2781] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:35.403026 6894 raft_consensus.cc:3032] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
W20260806 03:52:35.415710 3585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36468 (request call id 480) took 2312 ms (client timeout 1872 ms). Trace:
I20260806 03:52:35.416455 6894 raft_consensus.cc:729] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:35.416625 3585 rpcz_store.cc:269] 0806 03:52:33.103503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:35.414997 (+2311494us) service_pool.cc:214] Skipping call since client already timed out
0806 03:52:35.415300 (+ 303us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260806 03:52:35.418021 6894 consensus_queue.cc:260] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.419669 6894 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:35.422021 7086 log.cc:924] Time spent T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.929s user 0.002s sys 0.000s
I20260806 03:52:35.426908 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.428722 6490 consensus_queue.cc:1046] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:52:35.429942 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.430166 6490 raft_consensus.cc:3032] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:35.431025 2575 raft_consensus.cc:2396] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260806 03:52:35.430735 6490 raft_consensus.cc:729] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:35.432260 6915 consensus_queue.cc:1046] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:35.437201 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.427881 2577 raft_consensus.cc:2379] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 7: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:35.443429 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.445410 2376 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:35.444420 6815 raft_consensus.cc:686] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:35.445073 2379 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Advancing to term 4
I20260806 03:52:35.455176 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.458619 6815 consensus_queue.cc:237] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:35.463219 2429 leader_election.cc:334] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.800s (SENT)
I20260806 03:52:35.469471 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.469188 6794 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:35.473023 2369 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:35.478950 2429 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.482736 7038 raft_consensus.cc:3032] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:35.494244 7038 raft_consensus.cc:729] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:35.495653 7038 consensus_queue.cc:260] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.497750 7038 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:35.486506 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.500505 2375 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:35.483103 7006 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.006s
I20260806 03:52:35.482831 6794 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.504276 6490 consensus_queue.cc:260] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.486275 6770 raft_consensus.cc:2726] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:35.485960 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:35.506089 6490 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:35.506428 2377 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:35.514284 6861 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.518877 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40712 (request call id 495) took 2525 ms. Trace:
I20260806 03:52:35.519369 2372 rpcz_store.cc:276] 0806 03:52:32.993710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.610951 (+1617241us) service_pool.cc:227] Handling call
0806 03:52:35.380744 (+769793us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b5800003c98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a57726 00007f0d81a57900 00007f0d81a0ab50 00007f0d7b101807 00007f0d7b101792 00007f0d7b1016c1
0806 03:52:35.518768 (+138024us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29962112}
I20260806 03:52:35.492806 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.488093 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.523803 7050 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.525257 6841 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.526423 7050 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.528584 3594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36484 (request call id 344) took 3469 ms. Trace:
I20260806 03:52:35.529412 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2317) took 1618 ms. Trace:
I20260806 03:52:35.533527 6794 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:35.537707 7116 log.cc:924] Time spent T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260806 03:52:35.536839 2235 catalog_manager.cc:5495] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:35.527850 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.520958 2369 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:35.553287 3594 rpcz_store.cc:276] 0806 03:52:32.059487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:32.059636 (+ 149us) service_pool.cc:227] Handling call
0806 03:52:32.059997 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:32.100082 (+ 40085us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:32.100131 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:35.301231 (+3201100us) spinlock_profiling.cc:243] Waited 6.2 ms on lock 0x7b2c0017f298. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:52:35.324521 (+ 23290us) log.cc:842] Serialized 26 byte log entry
0806 03:52:35.446845 (+122324us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:35.446884 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:35.468911 (+ 22027us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:35.520652 (+ 51741us) raft_consensus.cc:1671] finished
0806 03:52:35.520682 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:35.528328 (+ 7646us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8079,"raft.queue_time_us":31510,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":24607,"spinlock_wait_cycles":14268160,"thread_start_us":115524,"threads_started":1,"wal-append.queue_time_us":171605}
I20260806 03:52:35.553367 2484 rpcz_store.cc:276] 0806 03:52:33.911400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:34.602367 (+690967us) service_pool.cc:227] Handling call
0806 03:52:34.621759 (+ 19392us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b58002d3f98. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79b65851 00007f0d79b65702 00007f0d82a55b09 00007f0d81991730 00007f0d81a5779b 00007f0d81a57821 00007f0d81a1b2b0 00007f0d81a067f9 00007f0d81a062a2 00007f0d7b0ec684
0806 03:52:34.622076 (+ 317us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:35.077069 (+454993us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:35.529100 (+452031us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54002a0dd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:35.529321 (+ 221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":482747904}
I20260806 03:52:35.570488 2373 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 10.
W20260806 03:52:35.577193 6780 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.494s user 0.017s sys 0.066s
I20260806 03:52:35.578073 7068 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:35.578186 6780 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.518s user 0.018s sys 0.067s
I20260806 03:52:35.580489 2576 raft_consensus.cc:2379] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:35.587965 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:35.594877 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.598908 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:35.599839 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.599787 2369 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:35.604982 2581 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260806 03:52:35.606084 2484 raft_consensus.cc:1226] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:35.608721 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.609027 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:35.609520 2579 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:35.610226 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.610128 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.608754 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.611505 2483 raft_consensus.cc:1799] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Handling vote request from an unknown peer fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.604629 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:35.609573 2373 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:35.612243 2379 raft_consensus.cc:1261] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:35.605458 2581 pending_rounds.cc:76] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 8
I20260806 03:52:35.616437 2483 raft_consensus.cc:2396] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260806 03:52:35.617192 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.614871 2581 pending_rounds.cc:97] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260806 03:52:35.623319 2584 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:35.613734 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.622282 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:35.626497 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:35.627943 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.629159 3542 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:35.626688 2235 catalog_manager.cc:5495] T ef53ef49663446658709042138ce12f7 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260806 03:52:35.633468 2331 leader_election.cc:334] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.872s (SENT)
I20260806 03:52:35.628855 2581 raft_consensus.cc:2866] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:35.636473 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.637162 2584 raft_consensus.cc:2379] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 9: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:35.619822 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.611959 6794 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260806 03:52:35.640831 7068 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:35.644786 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:35.649185 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 196953us
I20260806 03:52:35.635437 6907 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.650858 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:35.651937 2373 raft_consensus.cc:3032] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:35.652693 2373 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:35.654243 2372 raft_consensus.cc:1226] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 9. Current term is 10. Ops: []
I20260806 03:52:35.651340 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.657048 2577 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:35.662818 2372 raft_consensus.cc:1226] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:35.663908 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.664672 2372 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:35.664748 2235 catalog_manager.cc:5495] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:35.665470 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.665597 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.666155 2374 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 10.
I20260806 03:52:35.666327 2372 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:35.667114 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.667331 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.666595 2235 catalog_manager.cc:5495] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 3, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 3 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:35.668372 2372 raft_consensus.cc:2379] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 8: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
W20260806 03:52:35.672925 3547 connection.cc:232] RPC call timeout handler was delayed by -1.1474566711112857s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2716s for a response
I20260806 03:52:35.673380 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.674363 7050 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.674683 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:35.675387 2372 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:35.656750 6907 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.644860 7006 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:35.675045 7050 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:35.685405 2429 leader_election.cc:334] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.810s (SENT)
I20260806 03:52:35.685261 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.686273 2484 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260806 03:52:35.652186 6770 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.687676 2584 raft_consensus.cc:1261] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260806 03:52:35.689204 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.689918 2374 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:35.690425 2429 leader_election.cc:302] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.693020 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.694362 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.692318 6921 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.691409 7072 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:35.659657 7113 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:35.704638 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:35.704629 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.699371 2235 catalog_manager.cc:5495] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 5 to 7, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 7 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:35.686757 7006 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:35.704634 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.703951 3594 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:35.707713 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.708195 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.705997 7005 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:35.708566 3585 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:35.710201 2374 raft_consensus.cc:1226] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:35.708930 7005 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:35.712184 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 85335.7us
I20260806 03:52:35.706175 2373 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.700778 7111 raft_consensus.cc:2726] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260806 03:52:35.713857 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 91081.9us
I20260806 03:52:35.716221 2377 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 10.
I20260806 03:52:35.717334 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.716499 7111 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.720845 2373 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:35.721762 2331 leader_election.cc:334] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215): Timed out: RequestConsensusVote RPC to 127.0.183.4:33215 timed out after 1.913s (SENT)
I20260806 03:52:35.709707 2235 catalog_manager.cc:5495] T 42ea7d4af5d44cab89bf24f15376456e P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 5 to 7, leader changed from 608e57d72aae496aaf8613622153aa0c (127.0.183.2) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 7 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:35.718850 2379 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:35.722894 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.724296 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:35.710170 7006 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.720706 7111 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.724004 6794 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260806 03:52:35.697770 6921 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.743079 6815 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.696167 6770 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.743752 6815 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.745316 6770 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.746016 6915 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.743116 6921 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.746649 6915 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.747295 2576 raft_consensus.cc:1226] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 6. Current term is 7. Ops: []
I20260806 03:52:35.747402 2534 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:35.709717 7113 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.749289 6915 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:35.733090 3545 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.750543 2534 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:35.750837 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.734278 2584 raft_consensus.cc:3032] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:52:35.736945 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.749728 7113 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.753324 6907 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.760447 6921 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:52:35.760974 6373 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.706182 7072 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.765763 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:35.766069 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.766501 2483 raft_consensus.cc:3037] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:35.763088 6373 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.766888 3590 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:35.768812 6373 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.751199 2235 catalog_manager.cc:5495] T 5489408bdcc042f3919eae139ea82e7a P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 5 to 6, leader changed from cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4) to 608e57d72aae496aaf8613622153aa0c (127.0.183.2). New cstate: current_term: 6 leader_uuid: "608e57d72aae496aaf8613622153aa0c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:35.771024 2235 catalog_manager.cc:5495] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:35.775098 2235 catalog_manager.cc:5495] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 3 to 8, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 8 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:35.789004 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.789880 3585 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 10.
I20260806 03:52:35.790853 2370 raft_consensus.cc:1261] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260806 03:52:35.792178 2370 raft_consensus.cc:1226] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:35.792404 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:35.778015 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 99289.1us
W20260806 03:52:35.793751 2429 leader_election.cc:334] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.899s (SENT)
I20260806 03:52:35.794049 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.794214 2429 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.794317 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.778224 7072 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.779592 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:35.796312 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:35.796952 2474 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:35.795884 6907 raft_consensus.cc:2726] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 LEADER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:52:35.800449 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.800585 6841 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:35.801132 2484 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:35.801174 6841 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.802757 3547 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.781508 7005 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.782256 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.782866 7050 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.805289 7050 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:35.805895 2481 raft_consensus.cc:1261] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:52:35.783797 6780 consensus_queue.cc:1035] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260806 03:52:35.806052 2478 raft_consensus.cc:1261] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:35.796447 2376 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.809027 2331 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:35.793900 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:35.782449 2483 raft_consensus.cc:2396] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:52:35.787322 2235 catalog_manager.cc:5495] T 507574ca8b05468888fa8465ccdb2f80 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 6 to 9. New cstate: current_term: 9 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:35.764305 2372 raft_consensus.cc:2454] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
W20260806 03:52:35.811529 2429 leader_election.cc:334] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 2.096s (SENT)
W20260806 03:52:35.812148 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 61697.2us
I20260806 03:52:35.751618 2584 raft_consensus.cc:729] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:35.711252 6926 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:35.816619 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 60345.4us
I20260806 03:52:35.812947 2369 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 10.
I20260806 03:52:35.817849 2429 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.756204 7111 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.816429 6841 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:35.818816 6373 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:35.819056 6855 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:52:35.820565 2233 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33762 (request call id 225) took 32099 ms (client timeout 15000 ms). Trace:
I20260806 03:52:35.822530 3588 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 9. Current term is 10. Ops: []
I20260806 03:52:35.837708 6855 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:35.838167 6855 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.762985 6915 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260806 03:52:35.746059 6815 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:35.835017 2373 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260806 03:52:35.840906 2373 pending_rounds.cc:76] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 4
I20260806 03:52:35.841368 2373 pending_rounds.cc:97] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260806 03:52:35.841854 2373 raft_consensus.cc:2866] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:35.842907 2574 raft_consensus.cc:2379] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:35.833143 2376 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:35.843932 6926 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:35.839442 2233 rpcz_store.cc:269] 0806 03:52:03.619114 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:03.664271 (+ 45157us) service_pool.cc:227] Handling call
0806 03:52:35.712353 (+32048082us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0806 03:52:27.849886 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0806 03:52:28.152865 (+302979us) write_op.cc:432] Acquiring schema lock in shared mode
0806 03:52:28.152939 (+ 74us) write_op.cc:435] Acquired schema lock
0806 03:52:28.152980 (+ 41us) tablet.cc:660] Decoding operations
0806 03:52:28.154410 (+ 1430us) write_op.cc:620] Acquiring the partition lock for write op
0806 03:52:28.154517 (+ 107us) write_op.cc:641] Partition lock acquired for write op
0806 03:52:28.154553 (+ 36us) tablet.cc:683] Acquiring locks for 14 operations
0806 03:52:28.422678 (+268125us) tablet.cc:699] Row locks acquired
0806 03:52:28.422714 (+ 36us) write_op.cc:260] PREPARE: finished
0806 03:52:28.423061 (+ 347us) write_op.cc:270] Start()
0806 03:52:28.423279 (+ 218us) write_op.cc:276] Timestamp: P: 1785988348422949 usec, L: 0
0806 03:52:28.423317 (+ 38us) op_driver.cc:331] REPLICATION: starting
0806 03:52:28.517839 (+ 94522us) log.cc:842] Serialized 5478 byte log entry
0806 03:52:30.316867 (+1799028us) op_driver.cc:444] REPLICATION: finished
0806 03:52:30.557500 (+240633us) write_op.cc:301] APPLY: starting
0806 03:52:30.576617 (+ 19117us) tablet.cc:1365] starting BulkCheckPresence
0806 03:52:30.604766 (+ 28149us) tablet.cc:1368] finished BulkCheckPresence
0806 03:52:30.604904 (+ 138us) tablet.cc:1370] starting ApplyRowOperation cycle
0806 03:52:30.658208 (+ 53304us) tablet.cc:1381] finished ApplyRowOperation cycle
0806 03:52:30.658704 (+ 496us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0806 03:52:30.658775 (+ 71us) write_op.cc:312] APPLY: finished
0806 03:52:30.659375 (+ 600us) log.cc:842] Serialized 101 byte log entry
0806 03:52:30.659657 (+ 282us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:52:30.659897 (+ 240us) write_op.cc:454] Released schema lock
0806 03:52:30.660000 (+ 103us) write_op.cc:341] FINISH: Updating metrics
0806 03:52:30.671120 (+ 11120us) write_op.cc:489] Releasing partition, row and schema locks
0806 03:52:30.671185 (+ 65us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2527,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":160781,"apply.run_cpu_time_us":6915,"apply.run_wall_time_us":131989,"num_ops":14,"prepare.queue_time_us":278219,"prepare.run_cpu_time_us":39754,"prepare.run_wall_time_us":1992902,"raft.queue_time_us":85869,"raft.run_cpu_time_us":18416,"raft.run_wall_time_us":139058,"replication_time_us":1885305,"thread_start_us":1283925,"threads_started":4,"wal-append.queue_time_us":1178055,"wal-append.run_cpu_time_us":19495,"wal-append.run_wall_time_us":1976671}]]}
I20260806 03:52:35.848927 5510 consensus_queue.cc:1046] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:35.858981 6373 consensus_queue.cc:1046] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:35.850556 6862 consensus_queue.cc:1046] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:52:35.861673 6862 raft_consensus.cc:3032] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:35.849766 6915 consensus_queue.cc:1046] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 03:52:35.862053 6862 raft_consensus.cc:729] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:35.862737 7111 raft_consensus.cc:2726] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:35.862908 6915 raft_consensus.cc:3032] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:35.863443 6915 raft_consensus.cc:729] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:35.864176 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 68168.3us
I20260806 03:52:35.864490 6915 consensus_queue.cc:260] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:35.863185 6862 consensus_queue.cc:260] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.866691 6915 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.866976 6862 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:35.867249 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:35.867996 2478 raft_consensus.cc:3037] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.851166 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.855732 2370 raft_consensus.cc:1226] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260806 03:52:35.856624 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:35.872480 3585 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:35.859426 6815 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:35.873694 6841 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:35.853011 2534 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:35.874508 6841 consensus_queue.cc:1225] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 03:52:35.875861 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:35.876281 6926 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:35.876892 6926 raft_consensus.cc:482] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:35.877152 6384 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:35.877357 6926 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:35.877647 6384 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:35.877790 3584 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 9. Current term is 10. Ops: []
I20260806 03:52:35.878131 6384 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:35.884877 2584 consensus_queue.cc:260] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:35.847605 2537 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:35.888008 2537 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:35.889531 6894 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:35.890065 6894 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:35.890570 6894 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.890331 6490 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:52:35.891542 6490 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:35.892153 6490 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Advancing to term 11
W20260806 03:52:35.894119 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 91481.1us
I20260806 03:52:35.895816 3547 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.897064 6627 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.897275 7005 consensus_queue.cc:1035] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260806 03:52:35.904271 7005 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 03:52:35.905315 2476 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 03:52:35.907521 2429 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:35.909611 2476 raft_consensus.cc:1261] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:35.918298 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.912539 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.919260 2477 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:35.919368 2481 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:35.921660 7005 raft_consensus.cc:2726] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:52:35.933916 2474 raft_consensus.cc:2396] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 7 because replica has last-logged OpId of term: 5 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
W20260806 03:52:35.935312 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 50259.9us
W20260806 03:52:35.935230 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 54812.8us
I20260806 03:52:35.936807 2478 raft_consensus.cc:2454] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:35.939749 2534 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:35.943109 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:35.944105 2584 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:35.944612 2478 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 10: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:52:35.946151 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 56259.3us
W20260806 03:52:35.946278 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 87129.6us
I20260806 03:52:35.945047 2331 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:35.948043 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:35.948828 2477 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:35.955495 2537 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:35.960281 2474 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260806 03:52:35.960568 2332 leader_election.cc:334] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689): Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.130s (SENT)
I20260806 03:52:35.941826 7005 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:35.938766 2235 catalog_manager.cc:5495] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 6 to 7, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to fe3d17f4ad984b9088a268d58d87469c (127.0.183.3), now has a pending config: VOTER dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1), VOTER 608e57d72aae496aaf8613622153aa0c (127.0.183.2), VOTER fe3d17f4ad984b9088a268d58d87469c (127.0.183.3), VOTER cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 7 leader_uuid: "fe3d17f4ad984b9088a268d58d87469c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:35.968178 6489 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:35.968724 6841 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:35.969218 6841 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:35.969602 6841 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:35.965384 7111 consensus_queue.cc:1046] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
W20260806 03:52:35.971452 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 75782.2us
I20260806 03:52:35.971999 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:35.972262 2538 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:35.977190 2478 raft_consensus.cc:2362] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9. Re-sending same reply.
I20260806 03:52:35.981530 2429 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:35.983482 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 43787.9us
I20260806 03:52:35.984239 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:35.981403 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:35.965313 6815 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:35.987850 6815 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:35.988306 6815 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:35.989734 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.990784 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:35.993171 2477 raft_consensus.cc:2379] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:35.995003 2534 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:35.974299 7006 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.000525 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:35.970913 7050 raft_consensus.cc:2726] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:36.006316 2429 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:36.006731 7136 raft_consensus.cc:2741] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 03:52:36.007393 6896 raft_consensus.cc:2726] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:36.007993 5510 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:35.961949 6993 consensus_queue.cc:1046] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.040s
I20260806 03:52:35.964884 6373 consensus_queue.cc:1046] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260806 03:52:36.010196 2474 raft_consensus.cc:1226] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260806 03:52:36.008746 2534 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.021162 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:36.023345 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.025290 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.031705 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.033399 2235 catalog_manager.cc:5495] T 95f6f3de31d442cfa812406b89ee2fe3 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 3 to 4, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 4 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:36.035296 2370 raft_consensus.cc:2379] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:36.036648 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.034441 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.037597 2375 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:36.016580 5510 raft_consensus.cc:482] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.040563 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.042246 6489 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:36.042971 6489 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.043435 6489 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:36.043882 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.044517 6907 raft_consensus.cc:3032] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:36.044832 3584 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 10: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:36.044973 6907 raft_consensus.cc:729] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:36.046026 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:36.046782 3585 raft_consensus.cc:2362] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 10. Re-sending same reply.
I20260806 03:52:36.046844 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.048241 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.049161 3587 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:52:36.049266 3592 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:36.034782 6896 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:36.040563 5510 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.050328 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.051285 3587 raft_consensus.cc:2396] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 7 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 5.
I20260806 03:52:36.051349 3593 raft_consensus.cc:1261] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260806 03:52:36.046113 6907 consensus_queue.cc:260] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.052603 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.053514 6907 raft_consensus.cc:3037] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:36.054173 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.055054 3586 raft_consensus.cc:2379] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 10: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:36.048694 3588 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 6 LEARNER]: Advancing to term 9
I20260806 03:52:36.059702 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.047868 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.014138 7010 consensus_queue.cc:1046] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260806 03:52:36.066402 6489 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.001407 7005 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.069453 3588 raft_consensus.cc:1261] T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1 [term 9 LEARNER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260806 03:52:36.074304 7005 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:36.031152 7111 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:52:36.032390 7025 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:36.076814 7025 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.077349 7025 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:36.083761 2584 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 10.
I20260806 03:52:36.088868 6896 raft_consensus.cc:3032] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:52:36.088639 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.089423 6896 raft_consensus.cc:729] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:36.090086 2483 raft_consensus.cc:2379] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 3: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:36.090471 6896 consensus_queue.cc:260] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:36.088719 7068 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260806 03:52:36.092329 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.092995 2373 raft_consensus.cc:2454] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:36.095486 6373 raft_consensus.cc:3032] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:36.095890 6373 raft_consensus.cc:729] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:36.097010 6907 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:36.096984 6373 consensus_queue.cc:260] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.097975 6907 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.098624 6373 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:36.098770 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.076799 6855 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.101102 6907 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:36.102015 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 89200us
I20260806 03:52:36.102880 6855 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:36.103988 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.118902 3588 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:36.118717 6384 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.104393 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.121172 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.121686 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.122140 2483 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:36.122483 2474 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:36.124193 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.124888 2429 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:36.128906 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.129750 2474 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:36.129052 6907 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:36.108698 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.091867 6896 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:36.131893 2373 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 5.
I20260806 03:52:36.113430 7006 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:36.133857 2429 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:36.130877 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.134667 6907 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:36.137125 6907 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:36.137903 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:36.139928 7111 raft_consensus.cc:2781] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260806 03:52:36.140625 7111 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:36.141072 7111 raft_consensus.cc:3037] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Advancing to term 3
W20260806 03:52:36.143373 7038 consensus_meta.cc:287] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.645s user 0.012s sys 0.005s
I20260806 03:52:36.079778 6489 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:36.157958 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 117015us
I20260806 03:52:36.165401 6855 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.005s
I20260806 03:52:36.106531 5510 raft_consensus.cc:504] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.177479 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:36.179455 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 45758.4us
I20260806 03:52:36.180717 2429 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:36.184784 6861 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:36.185251 6861 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.185586 6861 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.188419 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.189189 2480 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:36.183132 2235 catalog_manager.cc:5495] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 4 to 6, leader changed from dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1) to cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4). New cstate: current_term: 6 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:36.207185 6862 raft_consensus.cc:3032] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:52:36.207747 6862 raft_consensus.cc:729] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:36.214514 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 193415us
I20260806 03:52:36.217494 3594 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:36.219658 2429 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:36.221091 2429 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:36.220902 6862 consensus_queue.cc:260] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.222266 6862 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:36.222430 2429 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:36.216043 2235 catalog_manager.cc:5495] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 1 to 8, leader changed from fe3d17f4ad984b9088a268d58d87469c (127.0.183.3) to dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1). New cstate: current_term: 8 leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } } }
I20260806 03:52:36.233047 6384 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:36.233664 2537 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.236056 7136 consensus_queue.cc:1035] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260806 03:52:36.239205 2235 catalog_manager.cc:5495] T f5efa489aaa94f9988b77e5efc032e5e P cf6042e85bef47c4aac9d064d91501d1 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "cf6042e85bef47c4aac9d064d91501d1" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260806 03:52:36.239813 2575 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:36.201813 7005 raft_consensus.cc:2781] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:36.217839 6627 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:36.261668 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:36.260102 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 235898us
I20260806 03:52:36.267594 7150 consensus_queue.cc:1035] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260806 03:52:36.268450 5510 leader_election.cc:288] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:36.268178 7025 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.285511 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.285715 6490 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.286615 2476 raft_consensus.cc:2379] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 10: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:36.286864 7025 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:36.270144 6915 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260806 03:52:36.271461 7068 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260806 03:52:36.258577 6384 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:36.256906 6841 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.290633 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:36.291285 3594 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:52:36.291754 6841 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:36.287529 7025 consensus_queue.cc:1035] T e9f040dd271944b9bf08554a90afa940 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.006s
I20260806 03:52:36.294555 6861 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.247193 7005 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:36.296343 7005 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:36.277315 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.278554 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.243860 7072 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:36.279042 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.247402 7113 raft_consensus.cc:2781] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:36.308357 7113 raft_consensus.cc:482] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:36.309031 7113 raft_consensus.cc:3037] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:36.282841 7148 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260806 03:52:36.280176 6855 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.020s
I20260806 03:52:36.280369 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.317997 6490 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:36.318236 6926 raft_consensus.cc:504] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.308723 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24cc4c26e48e41e3acb2121cf91b6df1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.324718 6815 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.332625 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.334609 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.373842 6926 leader_election.cc:288] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:36.335878 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.337177 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.375742 2476 raft_consensus.cc:2454] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:36.338637 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.377295 2484 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 5.
I20260806 03:52:36.339341 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.377595 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:36.378589 3585 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.388525 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.388866 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.390051 2484 raft_consensus.cc:2396] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 03:52:36.391834 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.404197 6815 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:36.406005 6894 consensus_meta.cc:287] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.515s user 0.018s sys 0.005s
I20260806 03:52:36.406777 6894 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.408944 2578 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:36.411513 3545 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.412518 7050 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:36.412899 7050 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.413239 7050 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:36.413398 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:36.415659 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2ddf38aaea54266a0a9e6100c48fc0d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.355427 2480 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:36.357195 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.417037 2484 raft_consensus.cc:2396] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260806 03:52:36.417476 2482 raft_consensus.cc:2379] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:36.357241 6384 raft_consensus.cc:686] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:36.418802 6384 consensus_queue.cc:237] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:36.420924 7153 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:36.421530 7153 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.421972 7153 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:52:36.355729 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 74007.3us
W20260806 03:52:36.426057 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 203689us
I20260806 03:52:36.427317 2429 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:36.433172 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 183894us
I20260806 03:52:36.358150 6861 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:36.368170 7111 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.374076 2581 raft_consensus.cc:2379] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:36.335069 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.415501 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:36.440482 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 96795us
I20260806 03:52:36.441488 2479 raft_consensus.cc:2454] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 11.
I20260806 03:52:36.445871 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.446548 2534 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:36.435634 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:36.440624 2477 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:36.354039 3594 raft_consensus.cc:2454] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 11.
W20260806 03:52:36.449944 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 180176us
I20260806 03:52:36.451143 2538 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:36.435487 6894 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:36.455894 2537 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.459352 2538 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.460796 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.464357 7164 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:52:36.466187 2479 raft_consensus.cc:2379] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 10: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:36.468675 2538 leader_election.cc:302] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.488080 6373 raft_consensus.cc:2781] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:36.489040 6373 raft_consensus.cc:686] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:36.490370 6373 consensus_queue.cc:237] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.445070 6862 consensus_queue.cc:1035] T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.023s
I20260806 03:52:36.515681 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:36.516552 2372 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 7
I20260806 03:52:36.518922 2479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ccb469db1664d3e941e7a360d048af2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.468606 6926 raft_consensus.cc:2781] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 11 FOLLOWER]: Leader election won for term 11
I20260806 03:52:36.460698 6861 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:52:36.467401 7148 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:36.531026 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.543826 2484 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.534392 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:36.536252 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.535760 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.545348 2369 raft_consensus.cc:2354] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9.
I20260806 03:52:36.537076 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.547375 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.539631 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:36.535715 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:36.546131 2474 raft_consensus.cc:3037] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:36.548323 2379 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 9: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:36.548334 2374 raft_consensus.cc:3037] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.546779 2377 raft_consensus.cc:1261] T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260806 03:52:36.550916 2376 raft_consensus.cc:1226] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 9. Current term is 10. Ops: []
I20260806 03:52:36.541121 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:36.551978 2370 raft_consensus.cc:3037] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.471385 6894 raft_consensus.cc:2781] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:36.548108 2368 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 4.
I20260806 03:52:36.560770 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.563499 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:36.564433 2577 raft_consensus.cc:3037] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.565114 3594 raft_consensus.cc:2379] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 10: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:36.567060 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:36.567960 2572 raft_consensus.cc:3037] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:36.571516 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.574414 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:36.575241 2373 raft_consensus.cc:3037] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:36.577097 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:36.579629 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:36.580835 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:36.573629 3542 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:36.500651 7005 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.585170 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:36.606110 2581 raft_consensus.cc:2396] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260806 03:52:36.541662 7148 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:36.585942 2481 raft_consensus.cc:2379] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 5: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:36.540351 6384 consensus_queue.cc:1035] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.063s
I20260806 03:52:36.607620 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:36.608196 2477 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:36.537221 6907 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.614570 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:36.582382 7072 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:36.638322 2368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:36.639439 2368 raft_consensus.cc:2379] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 9: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:36.639580 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:36.623785 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 158475us
I20260806 03:52:36.609092 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:36.607007 7164 raft_consensus.cc:686] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [term 10 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:36.607339 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:36.642760 2579 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.643035 3586 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.568665 3585 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:36.556290 7113 raft_consensus.cc:504] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.630287 7111 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:36.621209 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.618458 3542 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:36.616176 2574 pending_rounds.cc:76] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Aborting all ops after (but not including) 2
I20260806 03:52:36.659354 2429 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:36.659895 2574 pending_rounds.cc:97] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260806 03:52:36.664165 2574 raft_consensus.cc:2866] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:36.631978 2474 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:36.505122 7165 consensus_queue.cc:1035] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.187s
I20260806 03:52:36.677218 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.648399 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8a4475fdfd488b8b2a5456043cfbc5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.652177 2373 raft_consensus.cc:2454] T 0d63bf35a2e44ee881530227ba8e7e99 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 5.
I20260806 03:52:36.649452 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.569960 6861 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.063s
I20260806 03:52:36.674141 2372 raft_consensus.cc:2454] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:36.602485 6627 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:36.637439 2583 raft_consensus.cc:2379] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 9: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:36.660723 2482 raft_consensus.cc:2379] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:36.625113 7025 raft_consensus.cc:482] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:36.664404 2537 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.617446 7148 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:36.683990 2331 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.684835 2429 leader_election.cc:302] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:36.702378 7165 raft_consensus.cc:2781] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:36.702813 7165 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.703168 7165 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:52:36.704720 7162 log.cc:924] Time spent T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260806 03:52:36.699046 6780 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:52:36.706235 3584 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 10. Current term is 11. Ops: [8.8-10.9]
I20260806 03:52:36.706655 3588 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:36.707630 7164 consensus_queue.cc:237] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:36.708323 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.709594 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:36.685941 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 67625.6us
I20260806 03:52:36.665544 7163 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:36.710736 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d63bf35a2e44ee881530227ba8e7e99" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.723676 2429 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:36.687031 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:36.704530 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.715854 2370 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:36.726239 2474 raft_consensus.cc:3037] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.665518 7158 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:36.711372 2374 raft_consensus.cc:2454] T 7724acebda824fbcbf3d5bbc6a4b820c P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:36.704516 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.728893 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.754774 2481 raft_consensus.cc:2454] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 4.
I20260806 03:52:36.687932 6627 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:36.762735 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fb2162a24b741ebbc9732b674259c2d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.713349 7151 raft_consensus.cc:2781] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:36.766640 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.770521 2331 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:36.802618 2372 raft_consensus.cc:1261] T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:36.770004 732 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260806 03:52:36.807577 2374 raft_consensus.cc:1261] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260806 03:52:36.866770 2373 raft_consensus.cc:1261] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:36.846191 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.753382 6855 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:36.818421 2538 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:36.890901 2579 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:36.753720 7158 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.699280 7025 raft_consensus.cc:504] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:36.908788 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 198111us
I20260806 03:52:36.914681 3542 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:36.910076 2474 raft_consensus.cc:2454] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:36.917497 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:36.846109 7173 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260806 03:52:36.767447 7166 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:36.769052 6627 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:36.924636 2572 raft_consensus.cc:2454] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 9.
I20260806 03:52:36.923929 3586 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:36.944625 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:36.955757 2477 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:36.801556 6926 raft_consensus.cc:686] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 11 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:36.767189 6815 consensus_queue.cc:1046] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.171s
W20260806 03:52:36.967571 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 197106us
I20260806 03:52:36.976372 2331 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:36.904588 7156 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:36.973225 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:36.916833 7158 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:36.984889 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.972141 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.943737 7166 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:36.982529 2474 raft_consensus.cc:2454] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:37.008137 2477 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:36.967128 7005 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:36.972754 6780 raft_consensus.cc:686] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [term 10 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:36.756960 7163 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:37.029662 3586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:36.989239 6861 raft_consensus.cc:2726] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:52:37.041865 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:37.045286 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:37.046564 3594 raft_consensus.cc:2354] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
W20260806 03:52:37.010802 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 192487us
I20260806 03:52:36.999928 6862 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:37.000581 6907 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:36.995015 7166 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:37.025471 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:37.077494 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:37.039568 5510 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:37.080695 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:37.087006 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:37.076799 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 162256us
I20260806 03:52:37.027860 7113 leader_election.cc:288] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:37.111047 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:37.113766 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:37.103601 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:37.117187 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:37.116611 2477 raft_consensus.cc:2379] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:37.127599 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84da474ba5944a7a84d7c76403b92a63" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:37.127599 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:37.135639 2484 consensus_meta.cc:287] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.562s user 0.013s sys 0.000s
I20260806 03:52:37.136358 2484 raft_consensus.cc:2454] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:37.140178 2477 raft_consensus.cc:1226] T 84da474ba5944a7a84d7c76403b92a63 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 2. Current term is 3. Ops: []
I20260806 03:52:37.143922 2538 leader_election.cc:302] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:37.195393 2481 raft_consensus.cc:1261] T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260806 03:52:37.027644 7156 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:37.196058 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 52266.7us
I20260806 03:52:37.192689 2484 raft_consensus.cc:1226] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:37.207408 2477 raft_consensus.cc:1261] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260806 03:52:37.108496 6855 raft_consensus.cc:686] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [term 9 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:37.088197 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:37.201484 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:37.096669 6861 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260806 03:52:37.210441 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 209489us
I20260806 03:52:37.064831 6862 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:37.096392 7194 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 10 FOLLOWER]: Leader election won for term 10
W20260806 03:52:37.221190 2577 consensus_meta.cc:287] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.656s user 0.011s sys 0.003s
I20260806 03:52:37.223562 2331 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:37.227861 2577 raft_consensus.cc:2454] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 8.
I20260806 03:52:37.103811 7151 raft_consensus.cc:686] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:37.139223 6770 raft_consensus.cc:2741] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260806 03:52:37.230096 2331 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:37.233726 3591 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260806 03:52:37.236712 2331 leader_election.cc:302] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:37.198972 7175 raft_consensus.cc:2781] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader election won for term 7
I20260806 03:52:37.114849 5510 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:37.260989 2579 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:37.261147 2436 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
W20260806 03:52:37.273041 7181 log.cc:924] Time spent T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20260806 03:52:37.273926 7163 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:37.254477 6780 consensus_queue.cc:237] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:37.275861 7214 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:37.276583 7214 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:37.303680 2436 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:37.127682 6926 consensus_queue.cc:237] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:37.311931 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 213343us
I20260806 03:52:37.266417 7025 leader_election.cc:288] T 42ea7d4af5d44cab89bf24f15376456e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:37.269883 6815 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:37.241214 6861 consensus_queue.cc:1046] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.063s
I20260806 03:52:37.271095 7175 raft_consensus.cc:686] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:37.331643 3542 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:37.334865 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 54219.8us
I20260806 03:52:37.318080 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:37.367260 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 109336us
I20260806 03:52:37.278944 7159 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:37.265863 7194 raft_consensus.cc:686] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [term 10 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
W20260806 03:52:37.320770 7050 consensus_meta.cc:287] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.907s user 0.014s sys 0.003s
I20260806 03:52:37.313882 7113 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:37.384866 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 53369.5us
W20260806 03:52:37.376176 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 139560us
W20260806 03:52:37.387897 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 347925us
I20260806 03:52:37.345523 6993 raft_consensus.cc:2781] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:37.368580 6907 raft_consensus.cc:2741] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260806 03:52:37.389770 2436 leader_election.cc:302] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:37.344874 7025 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:37.434278 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 11 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:37.448611 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.019s sys 0.034s
W20260806 03:52:37.449956 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.578s user 0.020s sys 0.034s
W20260806 03:52:37.450464 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 579240us
I20260806 03:52:37.386832 6861 raft_consensus.cc:2781] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election won for term 9
I20260806 03:52:37.381246 7159 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:37.402627 2332 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.2:45689 timed out after 2.501s (SENT) [suppressed 48 similar messages]
W20260806 03:52:37.462498 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 59912.5us
W20260806 03:52:37.469496 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 79862.1us
I20260806 03:52:37.368172 6815 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:37.484041 2436 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:37.485620 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 87825us
I20260806 03:52:37.456655 7186 raft_consensus.cc:2726] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:37.493707 2436 leader_election.cc:302] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:37.495970 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 44866.7us
I20260806 03:52:37.469091 7163 raft_consensus.cc:2726] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:37.468470 7159 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:37.513367 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 125043us
I20260806 03:52:37.505420 2436 leader_election.cc:302] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:37.390712 7050 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:37.394554 7113 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:37.521677 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:37.379073 6855 consensus_queue.cc:237] T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:37.527364 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:37.525197 2436 leader_election.cc:398] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:37.528131 2477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:37.528611 2436 leader_election.cc:401] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20260806 03:52:37.530627 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.003s sys 0.011s
W20260806 03:52:37.531771 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.607s user 0.003s sys 0.011s
I20260806 03:52:37.533170 2332 leader_election.cc:302] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:37.538555 7177 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.377s user 0.001s sys 0.001s
I20260806 03:52:37.541782 2476 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2353) took 1025 ms. Trace:
I20260806 03:52:37.543133 6815 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:37.534699 7194 consensus_queue.cc:237] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:37.547384 7217 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:37.510499 7005 raft_consensus.cc:2726] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:37.503057 6907 raft_consensus.cc:2726] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260806 03:52:37.542248 2476 rpcz_store.cc:276] 0806 03:52:36.516553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.516723 (+ 170us) service_pool.cc:227] Handling call
0806 03:52:36.517280 (+ 557us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:36.524282 (+ 7002us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:36.524334 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:36.554907 (+ 30573us) log.cc:842] Serialized 52 byte log entry
0806 03:52:36.807302 (+252395us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:37.093448 (+286146us) log.cc:842] Serialized 14 byte log entry
0806 03:52:37.093789 (+ 341us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:37.093901 (+ 112us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:37.541459 (+447558us) raft_consensus.cc:1671] finished
0806 03:52:37.541485 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:37.541675 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":77741,"prepare.queue_time_us":420247,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"raft.queue_time_us":125699,"raft.run_cpu_time_us":358,"raft.run_wall_time_us":28603,"spinlock_wait_cycles":1135488,"thread_start_us":297236,"threads_started":2,"wal-append.queue_time_us":365425}
I20260806 03:52:37.559832 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:37.507903 6862 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:37.523056 7214 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:37.563511 3590 raft_consensus.cc:2354] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9.
I20260806 03:52:37.573755 2478 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:37.597021 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:37.598197 2484 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:37.476477 7203 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260806 03:52:37.600343 2538 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:37.603597 2538 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:37.385497 7151 consensus_queue.cc:237] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:37.576948 7148 consensus_meta.cc:287] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.849s user 0.013s sys 0.002s
I20260806 03:52:37.612679 7209 raft_consensus.cc:2781] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:37.612839 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:37.614727 7204 raft_consensus.cc:482] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:37.615505 7204 raft_consensus.cc:504] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:37.608088 2583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
I20260806 03:52:37.616935 2583 raft_consensus.cc:595] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
W20260806 03:52:37.639822 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 118240us
W20260806 03:52:37.653505 2574 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.014s sys 0.017s
W20260806 03:52:37.654158 2574 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.663s user 0.015s sys 0.017s
I20260806 03:52:37.574689 6861 raft_consensus.cc:686] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:37.605461 7025 raft_consensus.cc:686] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:37.606307 7156 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:37.566597 7217 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:37.605121 5510 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:37.515082 7175 consensus_queue.cc:237] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:37.619376 6907 consensus_queue.cc:1035] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260806 03:52:37.567598 7197 consensus_queue.cc:1035] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.390s
I20260806 03:52:37.919705 3585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36468 (request call id 500) took 1149 ms. Trace:
I20260806 03:52:37.753984 7208 consensus_queue.cc:1046] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.393s
W20260806 03:52:37.920845 7218 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.257s user 0.000s sys 0.001s
I20260806 03:52:37.639889 7148 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:37.712342 7238 raft_consensus.cc:2726] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:37.706476 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:37.928403 3584 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:37.928845 7148 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:37.667299 7163 consensus_queue.cc:1046] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.263s
I20260806 03:52:37.752144 7167 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8.
I20260806 03:52:37.930847 7163 raft_consensus.cc:3032] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:52:37.928040 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2467) took 1127 ms. Trace:
I20260806 03:52:37.738910 7025 consensus_queue.cc:237] T ead4b99d10d24d13ba7f07dee24f79e3 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:37.932958 7153 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.510s user 0.013s sys 0.006s
I20260806 03:52:37.823153 7072 consensus_queue.cc:1046] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.111s
I20260806 03:52:37.927371 2581 raft_consensus.cc:1226] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 9. Current term is 10. Ops: []
I20260806 03:52:37.933478 7153 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:37.934374 7072 raft_consensus.cc:3032] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Stepping down as leader of term 3
I20260806 03:52:37.934894 2483 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260806 03:52:37.934978 7072 raft_consensus.cc:729] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:37.719717 7158 consensus_meta.cc:287] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.733s user 0.016s sys 0.000s
I20260806 03:52:37.936946 7158 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:37.937718 7153 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:37.931842 2370 rpcz_store.cc:276] 0806 03:52:36.800817 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.800964 (+ 147us) service_pool.cc:227] Handling call
0806 03:52:36.801255 (+ 291us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:37.060457 (+259202us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:37.060512 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:37.098623 (+ 38111us) log.cc:842] Serialized 26 byte log entry
0806 03:52:37.458930 (+360307us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:37.459800 (+ 870us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:37.459923 (+ 123us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:37.927706 (+467783us) raft_consensus.cc:1671] finished
0806 03:52:37.927734 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:37.927943 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11161,"raft.queue_time_us":88866,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":12162,"thread_start_us":482461,"threads_started":2,"wal-append.queue_time_us":445015}
I20260806 03:52:37.804824 7203 raft_consensus.cc:2781] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260806 03:52:37.738606 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 79762.9us
I20260806 03:52:37.805440 7156 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:37.926056 7226 log.cc:924] Time spent T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260806 03:52:37.821393 7233 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:37.706259 7199 consensus_queue.cc:1046] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.474s
I20260806 03:52:37.817870 6815 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:52:37.751441 7187 log.cc:924] Time spent T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.203s user 0.000s sys 0.001s
I20260806 03:52:37.925746 3585 rpcz_store.cc:276] 0806 03:52:36.770535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.796261 (+ 25726us) service_pool.cc:227] Handling call
0806 03:52:36.934651 (+138390us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:37.042287 (+107636us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:37.049619 (+ 7332us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:37.226506 (+176887us) log.cc:842] Serialized 26 byte log entry
0806 03:52:37.370028 (+143522us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:37.387667 (+ 17639us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:37.396527 (+ 8860us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:37.896101 (+499574us) raft_consensus.cc:1671] finished
0806 03:52:37.896408 (+ 307us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:37.917970 (+ 21562us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4680,"raft.queue_time_us":98293,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":36709,"thread_start_us":60915,"threads_started":1,"wal-append.queue_time_us":464093}
I20260806 03:52:37.930014 2534 leader_election.cc:398] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9.
I20260806 03:52:38.117985 2534 leader_election.cc:401] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260806 03:52:37.939632 7158 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:38.068686 7234 raft_consensus.cc:979] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: : Instructing follower 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:52:38.009429 2436 leader_election.cc:302] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:37.883173 6993 raft_consensus.cc:686] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:38.012336 7006 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.502s user 0.006s sys 0.014s
I20260806 03:52:38.014309 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:37.982327 7204 leader_election.cc:288] T 940c72b441104eabbcb3ea23683268c1 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:37.679025 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:38.025538 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 501362us
I20260806 03:52:38.072903 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "940c72b441104eabbcb3ea23683268c1" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:37.821769 7206 log.cc:924] Time spent T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.424s user 0.002s sys 0.000s
W20260806 03:52:37.751278 7191 log.cc:924] Time spent T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.280s user 0.000s sys 0.002s
W20260806 03:52:37.753803 7213 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.166s user 0.001s sys 0.000s
W20260806 03:52:37.705230 6894 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.083s user 0.003s sys 0.008s
I20260806 03:52:37.931223 7163 raft_consensus.cc:729] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:38.087778 3542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.584s user 0.021s sys 0.029s
W20260806 03:52:38.089136 7194 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s user 0.021s sys 0.034s
W20260806 03:52:38.089282 2535 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.035s sys 0.061s
W20260806 03:52:38.168377 2535 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.632s user 0.035s sys 0.062s
I20260806 03:52:38.090422 3547 leader_election.cc:302] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:38.169605 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 79290.3us
W20260806 03:52:38.166460 3542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.663s user 0.022s sys 0.029s
I20260806 03:52:38.170634 3547 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:38.091239 7196 raft_consensus.cc:3032] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Stepping down as leader of term 7
W20260806 03:52:38.101593 2482 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:54234 (request call id 2406) took 1686 ms (client timeout 1516 ms). Trace:
W20260806 03:52:38.105876 3590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 508) took 1693 ms (client timeout 1562 ms). Trace:
I20260806 03:52:38.113054 2480 raft_consensus.cc:1226] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:38.113886 2377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 501) took 1475 ms. Trace:
I20260806 03:52:38.066547 6907 consensus_queue.cc:1035] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20260806 03:52:38.173403 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:38.119635 2584 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [term 11 LEADER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260806 03:52:37.847051 7050 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:37.719168 6373 raft_consensus.cc:2726] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260806 03:52:37.904251 7224 raft_consensus.cc:3032] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 LEADER]: Stepping down as leader of term 10
I20260806 03:52:37.945029 7203 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:38.172588 7194 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s user 0.021s sys 0.034s
I20260806 03:52:38.012815 7209 raft_consensus.cc:686] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 LEADER]: Becoming Leader. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
I20260806 03:52:37.750573 7178 raft_consensus.cc:2726] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:38.081517 6815 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:38.081053 7113 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:38.154143 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 207966us
I20260806 03:52:38.116668 7197 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.146535 7217 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:38.170922 3542 leader_election.cc:302] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:38.234790 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 64000us
I20260806 03:52:38.236264 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 450) took 1431 ms. Trace:
I20260806 03:52:38.173877 2377 rpcz_store.cc:276] 0806 03:52:36.638629 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.646448 (+ 7819us) service_pool.cc:227] Handling call
0806 03:52:36.665682 (+ 19234us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:36.955984 (+290302us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:36.956547 (+ 563us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:36.956622 (+ 75us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:36.956663 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:38.113617 (+1156954us) spinlock_profiling.cc:243] Waited 290 ms on lock 0x7b5400012554. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:38.113672 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:38.113820 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":667094912}
W20260806 03:52:38.173576 3590 rpcz_store.cc:269] 0806 03:52:36.412843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.413040 (+ 197us) service_pool.cc:227] Handling call
0806 03:52:38.105493 (+1692453us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54006cfcd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:38.105726 (+ 233us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2609705728}
W20260806 03:52:38.173089 2482 rpcz_store.cc:269] 0806 03:52:36.428889 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.437022 (+ 8133us) service_pool.cc:227] Handling call
0806 03:52:38.101182 (+1664160us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b540023ffd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:38.101433 (+ 251us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":485447552}
I20260806 03:52:38.236714 2474 rpcz_store.cc:276] 0806 03:52:36.804887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:37.000443 (+195556us) service_pool.cc:227] Handling call
0806 03:52:37.042178 (+ 41735us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:37.050063 (+ 7885us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:37.050103 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:37.050599 (+ 496us) log.cc:842] Serialized 26 byte log entry
0806 03:52:37.268134 (+217535us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:37.268196 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:37.268295 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:38.235989 (+967694us) raft_consensus.cc:1671] finished
0806 03:52:38.236015 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:38.236188 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":578,"raft.queue_time_us":791,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":6683,"thread_start_us":129405,"threads_started":1,"wal-append.queue_time_us":796693}
W20260806 03:52:38.179117 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 249286us
I20260806 03:52:38.181030 2332 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
W20260806 03:52:38.153373 7165 consensus_meta.cc:287] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.450s user 0.012s sys 0.008s
I20260806 03:52:38.023761 7156 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:38.166432 6894 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.559s user 0.003s sys 0.012s
I20260806 03:52:38.157521 7204 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:37.997589 5510 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.195281 7242 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.195791 7235 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.196768 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2442) took 1634 ms. Trace:
I20260806 03:52:38.120597 7072 consensus_queue.cc:260] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.198339 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2359) took 1660 ms. Trace:
I20260806 03:52:38.213348 7210 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:38.199194 7243 raft_consensus.cc:2726] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:37.883304 6861 consensus_queue.cc:237] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:38.184725 6627 consensus_meta.cc:287] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.205s user 0.018s sys 0.000s
I20260806 03:52:38.177953 7196 raft_consensus.cc:729] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:38.068053 7233 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:38.184231 7190 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.901s user 0.015s sys 0.021s
I20260806 03:52:38.185099 7229 raft_consensus.cc:2726] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:38.200562 7203 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:38.185397 7224 raft_consensus.cc:729] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: dff94100ffc5485f92cbe6a85addf0b2, State: Running, Role: LEADER
W20260806 03:52:38.242529 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 61743.8us
I20260806 03:52:38.210108 6926 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643) with higher term. Message: Invalid argument: T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9.
I20260806 03:52:38.199060 7234 raft_consensus.cc:1067] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Signalling peer 608e57d72aae496aaf8613622153aa0c to start an election
I20260806 03:52:38.256348 2376 raft_consensus.cc:1261] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260806 03:52:38.256507 2377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:38.221771 7158 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.222710 6815 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:38.246470 2479 rpcz_store.cc:276] 0806 03:52:36.537980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.541195 (+ 3215us) service_pool.cc:227] Handling call
0806 03:52:36.567348 (+ 26153us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:36.892279 (+324931us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:36.892332 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:36.943372 (+ 51040us) log.cc:842] Serialized 52 byte log entry
0806 03:52:37.200061 (+256689us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:37.200102 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:37.200197 (+ 95us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:38.197911 (+997714us) raft_consensus.cc:1671] finished
0806 03:52:38.197946 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:38.198169 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14095,"raft.queue_time_us":78343,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":23454,"thread_start_us":426420,"threads_started":2,"wal-append.queue_time_us":941699}
I20260806 03:52:38.246268 2574 rpcz_store.cc:276] 0806 03:52:36.562283 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.572198 (+ 9915us) service_pool.cc:227] Handling call
0806 03:52:36.615799 (+ 43601us) raft_consensus.cc:1457] Updating replica for 3 ops
0806 03:52:36.704841 (+ 89042us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:36.704876 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0806 03:52:36.705632 (+ 756us) log.cc:842] Serialized 78 byte log entry
0806 03:52:36.973644 (+268012us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:36.988724 (+ 15080us) log.cc:842] Serialized 14 byte log entry
0806 03:52:37.654699 (+665975us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:37.654829 (+ 130us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:38.196366 (+541537us) raft_consensus.cc:1671] finished
0806 03:52:38.196401 (+ 35us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:38.196667 (+ 266us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":940,"prepare.queue_time_us":15384,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":276502,"raft.run_cpu_time_us":429,"raft.run_wall_time_us":22183,"spinlock_wait_cycles":1670400,"thread_start_us":931306,"threads_started":2,"wal-append.queue_time_us":727705}
I20260806 03:52:38.242106 7165 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.251253 5510 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:38.274896 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 127128us
I20260806 03:52:38.240584 6894 raft_consensus.cc:686] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [term 9 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:38.213086 7050 consensus_queue.cc:1035] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
W20260806 03:52:38.337113 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 135829us
I20260806 03:52:38.257452 7208 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.279426 6627 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:38.270853 7238 raft_consensus.cc:2726] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260806 03:52:38.272884 7210 raft_consensus.cc:2726] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260806 03:52:38.351025 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 80824.6us
I20260806 03:52:38.353454 2369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 499) took 1730 ms. Trace:
I20260806 03:52:38.355655 2332 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:38.276547 6490 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:38.356530 2583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05e17a7474ab4829922b72cc624de950"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:47302
W20260806 03:52:38.322613 7190 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.121s user 0.016s sys 0.026s
I20260806 03:52:38.266345 7204 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.360318 6896 raft_consensus.cc:3032] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Stepping down as leader of term 2
W20260806 03:52:38.361012 7166 consensus_meta.cc:287] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.259s user 0.018s sys 0.001s
I20260806 03:52:38.369241 7158 raft_consensus.cc:504] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:38.267717 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 45987.5us
I20260806 03:52:38.381863 2537 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
W20260806 03:52:38.488848 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 107107us
I20260806 03:52:38.265064 7242 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.492662 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:38.482255 7238 raft_consensus.cc:2726] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:38.257061 7163 consensus_queue.cc:260] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.318822 7223 consensus_queue.cc:1035] T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.112s
I20260806 03:52:38.257218 7235 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.348429 7201 consensus_queue.cc:1035] T 940c72b441104eabbcb3ea23683268c1 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.190s
I20260806 03:52:38.352675 7209 consensus_queue.cc:237] T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.251056 6993 consensus_queue.cc:237] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.390120 7233 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:38.392014 7165 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:38.249958 7197 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.349169 7151 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.372244 6373 raft_consensus.cc:482] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.442723 5510 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:38.447659 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 92120.8us
I20260806 03:52:38.436556 7229 consensus_queue.cc:1035] T f327872ee26848bbaf04c03cc38edd54 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.077s
I20260806 03:52:38.446993 2369 rpcz_store.cc:276] 0806 03:52:36.622553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.623419 (+ 866us) service_pool.cc:227] Handling call
0806 03:52:36.623774 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:36.641987 (+ 18213us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:36.642020 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:36.646587 (+ 4567us) log.cc:842] Serialized 26 byte log entry
0806 03:52:36.944280 (+297693us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:36.944365 (+ 85us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:36.944492 (+ 127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:38.353175 (+1408683us) raft_consensus.cc:1671] finished
0806 03:52:38.353202 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:38.353387 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":9730,"thread_start_us":219218,"threads_started":1,"wal-append.queue_time_us":642152}
I20260806 03:52:38.493520 2577 raft_consensus.cc:2379] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 9: Already voted for candidate dff94100ffc5485f92cbe6a85addf0b2 in this term.
I20260806 03:52:38.387022 7252 raft_consensus.cc:482] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.508625 2584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2443) took 1946 ms. Trace:
W20260806 03:52:38.509769 2579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1205) took 1406 ms (client timeout 1653 ms). Trace:
I20260806 03:52:38.506856 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44204 (request call id 1203) took 1408 ms. Trace:
I20260806 03:52:38.513829 2572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:38.474793 7217 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.482987 6855 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.521626 3590 raft_consensus.cc:1226] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:38.481465 6896 raft_consensus.cc:729] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:38.525189 2583 raft_consensus.cc:595] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Received request to transfer leadership to TS 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:38.506317 7234 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:38.448011 2534 leader_election.cc:302] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
W20260806 03:52:38.532413 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 134820us
I20260806 03:52:38.533113 2332 leader_election.cc:302] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.561878 2332 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:38.510766 7210 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:38.435894 7196 consensus_queue.cc:260] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:38.535244 7151 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:38.480868 6490 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:38.435603 7208 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:38.547813 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 59335.5us
I20260806 03:52:38.548899 3545 leader_election.cc:302] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.552744 2584 rpcz_store.cc:276] 0806 03:52:36.562636 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.572110 (+ 9474us) service_pool.cc:227] Handling call
0806 03:52:36.608586 (+ 36476us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:38.177304 (+1568718us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:38.508192 (+330888us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b540028f5d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:38.508531 (+ 339us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3474150656}
W20260806 03:52:38.552842 2579 rpcz_store.cc:269] 0806 03:52:37.089592 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:37.141033 (+ 51441us) service_pool.cc:227] Handling call
0806 03:52:38.493240 (+1352207us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b540024ccd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:38.493392 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":103179520}
I20260806 03:52:38.577786 7204 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:38.578504 7204 raft_consensus.cc:2781] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:38.578894 7204 raft_consensus.cc:482] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:38.579207 7204 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:38.579414 6627 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:38.558013 3585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:38.564148 7234 raft_consensus.cc:482] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:38.580840 7234 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:38.581746 7197 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:38.582448 7197 raft_consensus.cc:2781] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260806 03:52:38.582630 7242 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:38.582825 7197 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:38.583174 2477 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 463) took 1050 ms. Trace:
I20260806 03:52:38.583282 7197 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:38.583858 7235 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:38.584504 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2352) took 2076 ms. Trace:
I20260806 03:52:38.552971 2575 rpcz_store.cc:276] 0806 03:52:37.084400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:37.085437 (+ 1037us) service_pool.cc:227] Handling call
0806 03:52:38.492702 (+1407265us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540013bfd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:38.492913 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2419894656}
I20260806 03:52:38.557119 2583 raft_consensus.cc:603] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260806 03:52:38.584949 2480 rpcz_store.cc:276] 0806 03:52:36.508482 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.508668 (+ 186us) service_pool.cc:227] Handling call
0806 03:52:38.103858 (+1595190us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:38.182589 (+ 78731us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:38.584267 (+401678us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b540023ffd0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac4a09 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:38.584420 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3667537792}
I20260806 03:52:38.525031 7163 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Advancing to term 10
W20260806 03:52:38.535902 7038 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.874s user 0.015s sys 0.030s
I20260806 03:52:38.549098 7156 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:38.486995 7166 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:38.570672 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 122746us
I20260806 03:52:38.595033 2534 leader_election.cc:302] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:38.583642 2477 rpcz_store.cc:276] 0806 03:52:37.532376 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:37.548807 (+ 16431us) service_pool.cc:227] Handling call
0806 03:52:37.549191 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:38.240127 (+690936us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:38.240161 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:38.245846 (+ 5685us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:38.245895 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:38.582895 (+337000us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b54001ac054. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:38.582952 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:38.583101 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1571528960}
I20260806 03:52:38.601845 2436 leader_election.cc:302] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:38.570523 7025 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.537801 6373 raft_consensus.cc:504] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.585165 7199 raft_consensus.cc:2781] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Leader election won for term 5
W20260806 03:52:38.590593 7038 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.934s user 0.015s sys 0.031s
I20260806 03:52:38.643162 2436 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.534801 6894 consensus_queue.cc:237] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.579201 7208 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:38.564581 6855 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.593891 7201 raft_consensus.cc:2726] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260806 03:52:38.639273 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 74236.9us
I20260806 03:52:38.628726 7242 raft_consensus.cc:2781] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:38.514354 7224 consensus_queue.cc:260] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:38.567350 7217 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:38.664539 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 102761us
I20260806 03:52:38.665495 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:38.639350 7165 raft_consensus.cc:2781] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:38.668947 2436 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.670488 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:38.593236 7214 consensus_queue.cc:1046] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260806 03:52:38.568852 7210 raft_consensus.cc:504] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:38.680084 3590 raft_consensus.cc:2354] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9.
I20260806 03:52:38.684448 2332 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.562495 7252 raft_consensus.cc:504] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.686797 7273 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:38.688248 7274 log.cc:924] Time spent T 04b1317a33c34953b84b5a8c6598368f P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
I20260806 03:52:38.688977 3591 raft_consensus.cc:1226] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Rejecting Update request from peer dff94100ffc5485f92cbe6a85addf0b2 for earlier term 10. Current term is 11. Ops: [8.8-10.9]
W20260806 03:52:38.689360 7276 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.129s user 0.001s sys 0.000s
W20260806 03:52:38.690347 7269 log.cc:924] Time spent T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.118s user 0.001s sys 0.000s
I20260806 03:52:38.683844 7068 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.691043 2436 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.691416 7136 consensus_queue.cc:1035] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260806 03:52:38.714149 3585 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:38.650947 7233 consensus_queue.cc:1035] T 24cc4c26e48e41e3acb2121cf91b6df1 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260806 03:52:38.654718 7199 raft_consensus.cc:686] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:38.720839 7199 consensus_queue.cc:237] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.722585 7217 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:38.648006 6896 consensus_queue.cc:260] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.704996 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8655b80017d7422082efde215e797b5b" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:38.725274 2370 raft_consensus.cc:2454] T 8655b80017d7422082efde215e797b5b P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 10.
I20260806 03:52:38.636355 7010 consensus_queue.cc:1035] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260806 03:52:38.693588 2369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:38.728492 2369 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:38.707369 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:38.729230 7164 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:38.729532 2376 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:38.708307 7175 raft_consensus.cc:3032] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:38.694257 2372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:38.730207 7175 raft_consensus.cc:729] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:38.730558 2372 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 4 FOLLOWER]: Advancing to term 6
I20260806 03:52:38.731285 7175 consensus_queue.cc:260] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.732957 7175 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:38.670476 7165 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:38.659574 7271 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:38.637883 7225 consensus_queue.cc:1035] T bd8a4475fdfd488b8b2a5456043cfbc5 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260806 03:52:38.636556 7025 raft_consensus.cc:504] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:38.716645 7254 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.685413 7224 raft_consensus.cc:3037] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:52:38.755255 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:38.693066 7242 raft_consensus.cc:686] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:38.683637 6907 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:38.656270 6770 consensus_queue.cc:1035] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260806 03:52:38.683521 7111 raft_consensus.cc:482] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.751417 2436 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.716769 2379 raft_consensus.cc:1261] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:52:38.755394 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:38.893664 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 142402us
I20260806 03:52:38.894192 2376 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:38.756342 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:38.718077 7068 raft_consensus.cc:504] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:38.757498 2575 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:38.917060 2482 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
W20260806 03:52:38.770542 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 103674us
I20260806 03:52:38.917830 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:38.740058 7165 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:38.918642 2370 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:38.895129 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:38.894989 2436 leader_election.cc:302] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.715080 7158 leader_election.cc:288] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:38.920805 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:38.922854 2436 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.921571 2373 raft_consensus.cc:2454] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:38.938150 3545 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:38.888681 7271 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:38.965161 2372 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:38.936815 2332 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:38.971879 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:38.972720 2373 raft_consensus.cc:2454] T fb91ba465b9845129828f07c4d452970 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 10.
I20260806 03:52:38.896867 7151 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
W20260806 03:52:38.961018 7159 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.421s user 0.010s sys 0.008s
I20260806 03:52:38.936929 6907 raft_consensus.cc:2726] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:38.991668 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 68915us
I20260806 03:52:38.940801 7214 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.005951 2436 leader_election.cc:302] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:38.957010 7229 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 03:52:38.969621 6815 consensus_meta.cc:287] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.583s user 0.007s sys 0.006s
W20260806 03:52:39.014819 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 96596.9us
I20260806 03:52:38.914567 7111 raft_consensus.cc:504] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:39.020357 2332 leader_election.cc:302] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:39.021874 7111 leader_election.cc:288] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:39.022490 7111 raft_consensus.cc:482] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.022992 7111 raft_consensus.cc:504] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:39.015115 2368 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37370 (request call id 502) took 2348 ms. Trace:
I20260806 03:52:39.025053 2368 rpcz_store.cc:276] 0806 03:52:36.656601 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.657721 (+ 1120us) service_pool.cc:227] Handling call
0806 03:52:36.679757 (+ 22036us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:37.248457 (+568700us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:37.248488 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:37.248531 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:37.248567 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:39.004262 (+1755695us) spinlock_profiling.cc:243] Waited 568 ms on lock 0x7b54000127d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:39.004339 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:39.004531 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1307252096}
W20260806 03:52:39.015440 7277 log.cc:924] Time spent T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.365s user 0.000s sys 0.002s
I20260806 03:52:39.033350 2332 leader_election.cc:302] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:39.000447 6855 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:39.010991 6861 raft_consensus.cc:2726] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:39.017197 7229 raft_consensus.cc:482] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:39.030766 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:39.031071 7166 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:39.083166 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:39.036912 6373 leader_election.cc:288] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:39.086877 2379 raft_consensus.cc:2454] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 10.
I20260806 03:52:39.061216 6770 raft_consensus.cc:2741] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260806 03:52:39.064412 6855 raft_consensus.cc:2726] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260806 03:52:39.066706 7229 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:39.050239 7290 raft_consensus.cc:2726] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:39.029604 6815 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:39.003752 7159 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:39.084514 7223 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.027246 7197 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:39.136005 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:39.103395 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 70132.3us
I20260806 03:52:39.028195 7214 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } attrs { replace: true } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:39.079278 7288 raft_consensus.cc:2726] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260806 03:52:39.133975 3594 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260806 03:52:38.758296 7243 consensus_queue.cc:1046] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.382s
I20260806 03:52:39.148861 7243 raft_consensus.cc:3032] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Stepping down as leader of term 6
I20260806 03:52:39.149319 7243 raft_consensus.cc:729] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
W20260806 03:52:39.150810 2331 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.652s user 0.014s sys 0.034s
I20260806 03:52:39.150424 7243 consensus_queue.cc:260] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:39.151837 7111 leader_election.cc:288] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:39.028975 7242 consensus_queue.cc:237] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:38.733821 7273 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:39.145588 7252 leader_election.cc:288] T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:39.146037 7293 log.cc:924] Time spent T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20260806 03:52:38.982089 7271 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:39.147455 7289 log.cc:924] Time spent T 14fa621f65da481d96d569096ded0a35 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.119s user 0.000s sys 0.001s
I20260806 03:52:38.758149 7254 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:39.151221 2331 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.653s user 0.014s sys 0.034s
I20260806 03:52:39.099287 7210 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:39.116952 7234 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.529s user 0.010s sys 0.005s
I20260806 03:52:39.182034 2374 raft_consensus.cc:2396] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260806 03:52:39.842936 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 1.34432e+06us
I20260806 03:52:39.849588 6926 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.850373 6926 raft_consensus.cc:504] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:39.850947 2332 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:39.852384 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:39.853370 2379 raft_consensus.cc:2396] T 750d734a677a4861bca565b69d29396e P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260806 03:52:39.857594 2379 raft_consensus.cc:1261] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:52:39.859309 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:39.844448 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 469) took 1179 ms. Trace:
I20260806 03:52:39.179137 7243 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:39.760381 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.005s sys 0.010s
W20260806 03:52:39.835884 7303 log.cc:924] Time spent T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.727s user 0.001s sys 0.000s
I20260806 03:52:39.860095 2379 raft_consensus.cc:2454] T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 10.
W20260806 03:52:39.865188 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.852s user 0.006s sys 0.011s
W20260806 03:52:39.879717 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 873963us
W20260806 03:52:39.478856 7269 log.cc:924] Time spent T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
W20260806 03:52:39.453789 7300 log.cc:924] Time spent T 940c72b441104eabbcb3ea23683268c1 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.189s user 0.001s sys 0.000s
W20260806 03:52:39.500942 7292 log.cc:924] Time spent T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.319s user 0.000s sys 0.001s
I20260806 03:52:39.818696 6993 raft_consensus.cc:482] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:39.909725 2476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:39.509135 6815 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:39.452133 6907 raft_consensus.cc:2726] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:52:39.923606 2474 raft_consensus.cc:1226] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 4. Current term is 5. Ops: []
I20260806 03:52:39.509960 7197 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:39.367944 7006 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.141490 7223 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:39.868238 7252 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:39.364238 7304 log.cc:924] Time spent T 1f9fc024596a478e80a8be24774725e5 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260806 03:52:39.222760 7068 leader_election.cc:288] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:39.196251 7025 leader_election.cc:288] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:39.919967 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:39.913829 6770 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.926498 2482 raft_consensus.cc:1261] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260806 03:52:39.967648 2482 pending_rounds.cc:76] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 7
I20260806 03:52:39.907187 7159 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Requested vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:39.967991 2482 pending_rounds.cc:97] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 5.8
I20260806 03:52:39.968322 2482 raft_consensus.cc:2866] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:39.928342 7308 raft_consensus.cc:482] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.848147 7234 raft_consensus.cc:504] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:39.979790 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:39.980589 2375 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:39.982892 2474 raft_consensus.cc:1261] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260806 03:52:39.983481 2474 pending_rounds.cc:76] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 3
I20260806 03:52:39.983736 2474 pending_rounds.cc:97] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260806 03:52:39.983995 2474 raft_consensus.cc:2866] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:39.915143 2476 raft_consensus.cc:3037] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Advancing to term 6
W20260806 03:52:39.915370 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 64522.3us
I20260806 03:52:39.985970 2332 leader_election.cc:302] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:39.906301 7113 raft_consensus.cc:482] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.924589 7288 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:39.937500 6926 leader_election.cc:288] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:39.993304 2478 raft_consensus.cc:1226] T 97ea60c38c204c989b4ca800e8478f18 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260806 03:52:39.997221 3586 raft_consensus.cc:1226] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:39.957008 7148 raft_consensus.cc:3032] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:40.006444 2379 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260806 03:52:40.003641 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:40.014782 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:40.009438 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:40.029152 2572 raft_consensus.cc:2379] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 in current term 8: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
W20260806 03:52:40.030442 7286 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.842s user 0.009s sys 0.014s
W20260806 03:52:40.030900 7286 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.982s user 0.009s sys 0.015s
I20260806 03:52:40.040457 3593 raft_consensus.cc:2379] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:40.043736 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:40.045106 2474 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:39.979153 3588 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36468 (request call id 501) took 2747 ms. Trace:
I20260806 03:52:39.994727 7159 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:39.947402 7006 raft_consensus.cc:504] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:39.960772 7252 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:40.011292 7148 raft_consensus.cc:729] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:39.929387 6993 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:40.087721 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:40.118810 2474 raft_consensus.cc:2362] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6. Re-sending same reply.
I20260806 03:52:40.049718 7273 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:39.985663 6770 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:40.100898 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 115034us
I20260806 03:52:40.137540 2332 leader_election.cc:302] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1
W20260806 03:52:40.079157 7203 consensus_meta.cc:287] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.708s user 0.011s sys 0.008s
I20260806 03:52:40.001549 7113 raft_consensus.cc:504] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:40.132853 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 272986us
I20260806 03:52:40.065052 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:40.144541 3592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:40.146301 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:40.147245 2474 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:40.148530 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42ea7d4af5d44cab89bf24f15376456e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:40.148641 2375 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:40.149318 2474 raft_consensus.cc:2454] T 42ea7d4af5d44cab89bf24f15376456e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 7.
I20260806 03:52:39.994154 7308 raft_consensus.cc:504] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:40.155246 7163 consensus_meta.cc:287] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.561s user 0.018s sys 0.000s
I20260806 03:52:40.179001 7323 raft_consensus.cc:482] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:40.199558 2581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 404) took 2767 ms. Trace:
I20260806 03:52:40.269101 2581 rpcz_store.cc:276] 0806 03:52:37.432616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:37.432779 (+ 163us) service_pool.cc:227] Handling call
0806 03:52:37.465780 (+ 33001us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:37.934236 (+468456us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:40.199100 (+2264864us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b540013bad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:40.199451 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1061089536}
I20260806 03:52:40.200222 2482 raft_consensus.cc:1261] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:40.206975 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:40.085700 7159 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.271278 2375 raft_consensus.cc:2454] T 0cb0e9b9f35b4cd488afa1ae281597bc P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
W20260806 03:52:40.278661 2373 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.972s user 0.013s sys 0.011s
I20260806 03:52:40.296811 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:40.299862 2373 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.993s user 0.013s sys 0.011s
I20260806 03:52:40.179075 7327 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:40.301088 2375 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
W20260806 03:52:40.305873 3542 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.707s user 0.000s sys 0.023s
W20260806 03:52:40.316359 3542 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.717s user 0.000s sys 0.024s
I20260806 03:52:40.308944 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb91ba465b9845129828f07c4d452970" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:40.324117 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 175010us
W20260806 03:52:40.347467 2369 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.003s sys 0.020s
W20260806 03:52:40.348343 2369 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s user 0.004s sys 0.021s
I20260806 03:52:40.366531 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:40.143657 7203 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.275871 7242 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:40.285092 7153 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:40.440161 2537 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.587s user 0.008s sys 0.014s
W20260806 03:52:40.440716 2537 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.587s user 0.008s sys 0.014s
I20260806 03:52:40.172760 7273 consensus_queue.cc:1046] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.193s
I20260806 03:52:40.305426 7271 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:40.365231 7163 raft_consensus.cc:482] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
W20260806 03:52:40.459640 7175 consensus_meta.cc:287] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.726s user 0.014s sys 0.007s
I20260806 03:52:40.449131 7153 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.536896 7335 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:40.537719 7335 raft_consensus.cc:504] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.540195 7006 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:40.541426 7339 log.cc:924] Time spent T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c: Append to log took a long time: real 0.238s user 0.000s sys 0.001s
W20260806 03:52:40.542797 7315 log.cc:924] Time spent T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.403s user 0.001s sys 0.000s
I20260806 03:52:40.442800 7254 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:40.540092 7312 log.cc:924] Time spent T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.090s user 0.000s sys 0.001s
I20260806 03:52:40.285739 7323 raft_consensus.cc:504] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.512121 7223 leader_election.cc:288] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:40.564087 2484 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 470) took 1898 ms. Trace:
I20260806 03:52:40.621636 2484 rpcz_store.cc:276] 0806 03:52:38.665735 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.665898 (+ 163us) service_pool.cc:227] Handling call
0806 03:52:38.682833 (+ 16935us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:38.897339 (+214506us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:38.916558 (+ 19219us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:38.916614 (+ 56us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:38.916656 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:40.563767 (+1647111us) spinlock_profiling.cc:243] Waited 214 ms on lock 0x7b54002a12d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:40.563828 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:40.563986 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":492623232}
I20260806 03:52:40.601801 2583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:47302 (request call id 20) took 2324 ms. Trace:
I20260806 03:52:40.623467 2583 rpcz_store.cc:276] 0806 03:52:38.277495 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.338978 (+ 61483us) service_pool.cc:227] Handling call
0806 03:52:40.601568 (+2262590us) spinlock_profiling.cc:243] Waited 72.2 ms on lock 0x7b540004ae54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cabfbc3 00007f0d81a0981a 00007f0d7b0fcd04 00007f0d7b0fcc92 00007f0d7b0fcbc1 00007f0d7b0fcb3c 00007f0d7b0fbdf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:40.601699 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":166055168}
I20260806 03:52:40.598744 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7724acebda824fbcbf3d5bbc6a4b820c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:40.602667 3590 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 516) took 1949 ms (client timeout 1562 ms). Trace:
W20260806 03:52:40.625353 2476 consensus_meta.cc:287] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 0.640s user 0.014s sys 0.000s
W20260806 03:52:40.625226 3590 rpcz_store.cc:269] 0806 03:52:38.652783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.652947 (+ 164us) service_pool.cc:227] Handling call
0806 03:52:40.602437 (+1949490us) spinlock_profiling.cc:243] Waited 8.1 ms on lock 0x7b54006cfcd4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:40.602582 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":18628864}
I20260806 03:52:40.619419 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2360) took 4052 ms. Trace:
I20260806 03:52:40.562849 2481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d90046b84bc4833ac8950bd501840d7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:40.626986 2483 rpcz_store.cc:276] 0806 03:52:36.567169 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.581407 (+ 14238us) service_pool.cc:227] Handling call
0806 03:52:36.616238 (+ 34831us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:37.935654 (+1319416us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:40.619038 (+2683384us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001de2d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:40.619338 (+ 300us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3031815936}
I20260806 03:52:40.637044 7335 leader_election.cc:288] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:40.637315 2483 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd"
dest_uuid: "608e57d72aae496aaf8613622153aa0c"
from {username='slave'} at 127.0.0.1:54220
I20260806 03:52:40.637740 7335 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:40.637969 2483 raft_consensus.cc:482] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 03:52:40.638367 2483 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:40.638276 7335 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:40.641764 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 54442.4us
I20260806 03:52:40.625907 2476 raft_consensus.cc:2454] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
W20260806 03:52:40.657241 3592 consensus_meta.cc:287] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 0.506s user 0.014s sys 0.000s
I20260806 03:52:40.657743 3592 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:40.369186 7148 consensus_queue.cc:260] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.452527 7242 raft_consensus.cc:504] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.596937 7323 leader_election.cc:288] T 8db383c38ecd45f2b2cb8c46ff360638 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:40.563426 7271 raft_consensus.cc:482] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:40.679028 2484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:40.683815 7313 log.cc:924] Time spent T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.596s user 0.001s sys 0.000s
I20260806 03:52:40.372538 7327 raft_consensus.cc:504] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:40.685271 7350 raft_consensus.cc:3032] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Stepping down as leader of term 7
I20260806 03:52:40.686086 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:40.706885 2579 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:40.683568 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:40.554108 7234 leader_election.cc:288] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:40.529232 7163 raft_consensus.cc:504] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.600865 6490 leader_election.cc:288] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:40.696183 2584 raft_consensus.cc:1261] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Refusing update from remote peer 608e57d72aae496aaf8613622153aa0c: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260806 03:52:40.701449 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7a939d2320d4a478b95733e1579e2b3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:40.701666 2581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:40.701925 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27aa6efa6253470d9fb619f5536e8795" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:40.723738 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
W20260806 03:52:40.680473 7204 consensus_meta.cc:287] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 2.101s user 0.008s sys 0.011s
I20260806 03:52:40.715878 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6364534d89cb43079c8dafda6b5b8875" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:40.762681 2579 raft_consensus.cc:1226] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260806 03:52:40.744294 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b91e8fd3fd1340d2bd8db09b77bd7b0f" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:40.744867 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:40.768953 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:40.769863 2577 raft_consensus.cc:3037] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:40.677433 7223 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:40.694365 7298 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:40.705287 7323 raft_consensus.cc:482] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:40.794070 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:40.799489 3594 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.016s sys 0.068s
W20260806 03:52:40.780233 2478 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s user 0.024s sys 0.061s
I20260806 03:52:40.717684 6993 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:40.706265 7350 raft_consensus.cc:729] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:40.798758 2483 raft_consensus.cc:504] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.804581 2584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:40.816135 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 49819us
W20260806 03:52:40.807924 3594 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.655s user 0.017s sys 0.069s
W20260806 03:52:40.808035 2478 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s user 0.024s sys 0.061s
I20260806 03:52:40.820044 2584 raft_consensus.cc:2454] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 7.
I20260806 03:52:40.726017 7308 leader_election.cc:288] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:40.828147 2577 raft_consensus.cc:2454] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
I20260806 03:52:40.726658 7252 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:40.830222 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:40.761623 7072 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Advancing to term 4
W20260806 03:52:40.778512 7312 log.cc:924] Time spent T 3fb2162a24b741ebbc9732b674259c2d P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.083s user 0.000s sys 0.001s
I20260806 03:52:40.764417 7234 raft_consensus.cc:482] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:40.764081 6490 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:40.736181 7355 raft_consensus.cc:482] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:40.718668 7271 raft_consensus.cc:504] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:40.773087 7204 raft_consensus.cc:504] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:41.303813 2483 leader_election.cc:288] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:40.948855 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 471) took 2246 ms. Trace:
W20260806 03:52:41.275393 2374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 2487) took 2381 ms (client timeout 3134 ms). Trace:
I20260806 03:52:40.065490 3588 rpcz_store.cc:276] 0806 03:52:37.231613 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:37.233816 (+ 2203us) service_pool.cc:227] Handling call
0806 03:52:37.272611 (+ 38795us) raft_consensus.cc:1457] Updating replica for 2 ops
0806 03:52:37.273223 (+ 612us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:37.273290 (+ 67us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0806 03:52:38.577170 (+1303880us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b2c0017f298. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:52:38.577900 (+ 730us) log.cc:842] Serialized 52 byte log entry
0806 03:52:38.667441 (+ 89541us) raft_consensus.cc:1631] Marking committed up to 3
0806 03:52:38.678002 (+ 10561us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:38.682628 (+ 4626us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:39.938245 (+1255617us) raft_consensus.cc:1671] finished
0806 03:52:39.938288 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:39.971352 (+ 33064us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1284,"raft.queue_time_us":111194,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":13598,"spinlock_wait_cycles":32499968,"thread_start_us":73352,"threads_started":1,"wal-append.queue_time_us":337709}
I20260806 03:52:41.126159 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f327872ee26848bbaf04c03cc38edd54" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:40.773365 6770 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:41.311815 2577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:41.310573 2575 raft_consensus.cc:2379] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c in current term 6: Already voted for candidate fe3d17f4ad984b9088a268d58d87469c in this term.
I20260806 03:52:41.319455 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:54220 (request call id 2389) took 1176 ms. Trace:
I20260806 03:52:41.315212 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "460a21786a7449039551dc0f1ffd04e2" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:41.030004 2429 leader_election.cc:334] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.940s (SENT)
I20260806 03:52:41.425766 2483 rpcz_store.cc:276] 0806 03:52:40.134894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:40.637148 (+502254us) service_pool.cc:227] Handling call
0806 03:52:41.311047 (+673899us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:40.950971 2436 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.881s user 0.008s sys 0.004s
W20260806 03:52:41.424422 2374 rpcz_store.cc:269] 0806 03:52:38.894563 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.894740 (+ 177us) service_pool.cc:227] Handling call
0806 03:52:41.274453 (+2379713us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b540014ee54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:41.275292 (+ 839us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":602700672}
W20260806 03:52:41.393702 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 296205us
I20260806 03:52:41.429023 2429 leader_election.cc:302] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:41.364610 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56018 (request call id 410) took 2872 ms. Trace:
I20260806 03:52:41.424937 2480 rpcz_store.cc:276] 0806 03:52:38.702591 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.702805 (+ 214us) service_pool.cc:227] Handling call
0806 03:52:38.703308 (+ 503us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:38.746171 (+ 42863us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:38.746216 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:38.746261 (+ 45us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:38.746293 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:40.948355 (+2202062us) spinlock_profiling.cc:243] Waited 42.6 ms on lock 0x7b54000afad4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:40.948448 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:40.948730 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98063616}
I20260806 03:52:41.429778 2574 rpcz_store.cc:276] 0806 03:52:38.494149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.533832 (+ 39683us) service_pool.cc:227] Handling call
0806 03:52:38.560771 (+ 26939us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:38.631525 (+ 70754us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:52:38.631577 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:38.631642 (+ 65us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:38.631688 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:41.364275 (+2732587us) spinlock_profiling.cc:243] Waited 37.6 ms on lock 0x7b54003a02d4. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:41.364336 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:41.364524 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32528,"raft.queue_time_us":33572,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":7020,"spinlock_wait_cycles":86499584}
W20260806 03:52:40.850132 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 191222us
I20260806 03:52:41.273823 2221 maintenance_manager.cc:419] P e1fd51235035487c913ffce700debedb: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033380
I20260806 03:52:40.834836 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:41.427960 2436 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.358s user 0.021s sys 0.026s
W20260806 03:52:41.363682 7229 consensus_meta.cc:287] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 2.245s user 0.016s sys 0.005s
I20260806 03:52:41.315697 2478 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2387) took 1296 ms. Trace:
I20260806 03:52:41.451956 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:41.347736 2537 leader_election.cc:398] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Vote denied by peer cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215) with higher term. Message: Invalid argument: T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9.
I20260806 03:52:41.453043 2370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2488) took 2526 ms. Trace:
I20260806 03:52:41.448710 2483 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56244 (request call id 477) took 1080 ms. Trace:
W20260806 03:52:41.453280 2537 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 105812us
I20260806 03:52:41.453685 2483 rpcz_store.cc:276] 0806 03:52:40.368042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:41.437043 (+1069001us) service_pool.cc:227] Handling call
0806 03:52:41.437511 (+ 468us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:41.447811 (+ 10300us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:41.448335 (+ 524us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:41.448391 (+ 56us) raft_consensus.cc:1631] Marking committed up to 9
0806 03:52:41.448442 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:41.448511 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:41.448640 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:41.454759 2578 raft_consensus.cc:2396] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260806 03:52:41.447717 6780 consensus_queue.cc:1046] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.004s
I20260806 03:52:40.804721 7113 leader_election.cc:288] T 8390916b6ac84b59b3943bf835a36a4c P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:41.002466 7203 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:40.850082 3591 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.406s user 0.066s sys 0.073s
W20260806 03:52:41.044924 7313 log.cc:924] Time spent T 24cc4c26e48e41e3acb2121cf91b6df1 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20260806 03:52:41.446727 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 416768us
W20260806 03:52:41.259392 7340 log.cc:924] Time spent T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.163s user 0.000s sys 0.000s
W20260806 03:52:41.279006 7165 consensus_meta.cc:287] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 2.330s user 0.019s sys 0.005s
I20260806 03:52:41.463653 2578 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2aa3fe3cb0444d6ea2df79f78a0d7543"
dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
mode: GRACEFUL
new_leader_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
from {username='slave'} at 127.0.0.1:47302
W20260806 03:52:41.449187 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 1.3793e+06us
I20260806 03:52:41.464535 2578 raft_consensus.cc:595] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Received request to transfer leadership to TS dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:41.465152 2578 raft_consensus.cc:603] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260806 03:52:41.257367 7267 log.cc:924] Time spent T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 2.691s user 0.025s sys 0.006s
I20260806 03:52:41.455638 2575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:41.260293 7335 leader_election.cc:288] T 507574ca8b05468888fa8465ccdb2f80 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:41.456311 2480 raft_consensus.cc:1226] T 01b082125feb462d9ba064ed2acf4dfd P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9. Ops: []
I20260806 03:52:41.455827 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97ea60c38c204c989b4ca800e8478f18" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
W20260806 03:52:41.261740 7224 consensus_meta.cc:287] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 2.502s user 0.018s sys 0.001s
I20260806 03:52:40.816095 7223 raft_consensus.cc:504] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:40.802918 7353 raft_consensus.cc:482] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260806 03:52:41.459204 3591 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.081s user 0.158s sys 0.180s
I20260806 03:52:41.453493 2370 rpcz_store.cc:276] 0806 03:52:38.926794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.927129 (+ 335us) service_pool.cc:227] Handling call
0806 03:52:38.956487 (+ 29358us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:38.989935 (+ 33448us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:38.989988 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:39.009174 (+ 19186us) log.cc:842] Serialized 26 byte log entry
0806 03:52:39.840882 (+831708us) raft_consensus.cc:1631] Marking committed up to 2
0806 03:52:39.879800 (+ 38918us) log.cc:842] Serialized 14 byte log entry
0806 03:52:39.921383 (+ 41583us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:39.921466 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:41.452793 (+1531327us) raft_consensus.cc:1671] finished
0806 03:52:41.452816 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:41.452969 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41279,"prepare.queue_time_us":38396,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":5347,"raft.queue_time_us":63124,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":26345,"spinlock_wait_cycles":25331968,"thread_start_us":415478,"threads_started":1,"wal-append.queue_time_us":959743}
I20260806 03:52:41.097198 7352 raft_consensus.cc:482] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:41.388717 7242 leader_election.cc:288] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:41.355769 7204 leader_election.cc:288] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 9 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:41.452512 2478 rpcz_store.cc:276] 0806 03:52:40.019302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:40.075965 (+ 56663us) service_pool.cc:227] Handling call
0806 03:52:40.122488 (+ 46523us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:40.122835 (+ 347us) raft_consensus.cc:1520] Early marking committed up to index 4
0806 03:52:40.151570 (+ 28735us) log.cc:842] Serialized 14 byte log entry
0806 03:52:41.299346 (+1147776us) log.cc:842] Serialized 14 byte log entry
0806 03:52:41.315205 (+ 15859us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:41.315290 (+ 85us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:41.315360 (+ 70us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:41.315454 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:41.315619 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38666,"prepare.queue_time_us":28670,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":68,"raft.queue_time_us":66865,"raft.run_cpu_time_us":444,"raft.run_wall_time_us":74362,"thread_start_us":1146837,"threads_started":2,"wal-append.queue_time_us":1249022}
I20260806 03:52:41.447532 2534 leader_election.cc:302] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:41.434949 2368 raft_consensus.cc:1226] T 27aa6efa6253470d9fb619f5536e8795 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260806 03:52:41.468129 3543 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.897s user 0.002s sys 0.011s
I20260806 03:52:41.469488 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:41.470331 7366 raft_consensus.cc:482] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:41.471004 7308 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:41.466437 2429 connection.cc:232] RPC call timeout handler was delayed by -1.069539865013212s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15910000000000002s for a response
I20260806 03:52:41.471524 7113 raft_consensus.cc:482] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:40.817631 7298 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.473585 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1"
I20260806 03:52:41.477240 2578 raft_consensus.cc:1226] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260806 03:52:41.344818 6490 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.478348 2372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2491) took 2462 ms. Trace:
I20260806 03:52:41.045336 7350 consensus_queue.cc:260] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:41.482079 7243 consensus_meta.cc:287] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c: Time spent flushing consensus metadata: real 1.606s user 0.008s sys 0.014s
I20260806 03:52:41.483587 2575 raft_consensus.cc:2454] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:41.451768 6770 raft_consensus.cc:2726] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:41.485260 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8390916b6ac84b59b3943bf835a36a4c" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:41.463589 7374 raft_consensus.cc:482] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:41.487354 7335 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:41.489765 2377 raft_consensus.cc:1261] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260806 03:52:41.265089 7327 leader_election.cc:288] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:40.831497 7323 raft_consensus.cc:504] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:41.353570 7185 consensus_queue.cc:1035] T 39115846d5144463b3a3a8e09e8b4500 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.164s
I20260806 03:52:41.489058 7163 leader_election.cc:288] T 24cc4c26e48e41e3acb2121cf91b6df1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:41.328377 7234 raft_consensus.cc:504] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:41.742590 3590 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:41.743666 2377 pending_rounds.cc:76] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Aborting all ops after (but not including) 8
I20260806 03:52:41.597538 2574 raft_consensus.cc:1226] T 7724acebda824fbcbf3d5bbc6a4b820c P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 7. Current term is 8. Ops: [7.6-7.6]
W20260806 03:52:41.593813 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 121535us
I20260806 03:52:41.744606 2372 rpcz_store.cc:276] 0806 03:52:39.016242 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:39.016407 (+ 165us) service_pool.cc:227] Handling call
0806 03:52:39.016873 (+ 466us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:39.028679 (+ 11806us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:39.028731 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:39.060614 (+ 31883us) log.cc:842] Serialized 26 byte log entry
0806 03:52:39.460399 (+399785us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:39.460913 (+ 514us) log.cc:842] Serialized 14 byte log entry
0806 03:52:40.136538 (+675625us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:40.136670 (+ 132us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:41.363072 (+1226402us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b1800441ac0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:41.478007 (+114935us) raft_consensus.cc:1671] finished
0806 03:52:41.478133 (+ 126us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:41.478281 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":480605,"prepare.queue_time_us":256,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":519190,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":3936,"spinlock_wait_cycles":28284800,"thread_start_us":584804,"threads_started":2,"wal-append.queue_time_us":885215}
I20260806 03:52:41.489970 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "750d734a677a4861bca565b69d29396e" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:41.744335 3543 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.301s user 0.007s sys 0.019s
I20260806 03:52:41.493836 2576 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44194 (request call id 2445) took 4930 ms. Trace:
I20260806 03:52:41.744258 2377 pending_rounds.cc:97] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260806 03:52:41.744145 3591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36474 (request call id 518) took 2664 ms. Trace:
I20260806 03:52:41.751411 2377 raft_consensus.cc:2866] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:41.743086 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1215) took 1030 ms. Trace:
I20260806 03:52:41.753823 2578 rpcz_store.cc:276] 0806 03:52:40.712603 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:41.476498 (+763895us) service_pool.cc:227] Handling call
0806 03:52:41.476946 (+ 448us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:41.742737 (+265791us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:41.743011 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:41.743587 2429 connection.cc:232] RPC call timeout handler was delayed by -1.0529797980561852s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22410000000000002s for a response
I20260806 03:52:41.328095 7355 raft_consensus.cc:504] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.769965 2574 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1216) took 1056 ms. Trace:
I20260806 03:52:41.805830 2483 raft_consensus.cc:2379] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 6: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:41.805877 2481 raft_consensus.cc:2396] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260806 03:52:41.751223 2576 rpcz_store.cc:276] 0806 03:52:36.563285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:36.563407 (+ 122us) service_pool.cc:227] Handling call
0806 03:52:36.580585 (+ 17178us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:36.913802 (+333217us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:36.913840 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:38.116736 (+1202896us) spinlock_profiling.cc:243] Waited 2.29 ms on lock 0x7b2c001765a8. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7b4a8e11 00007f0d7b4b8442 00007f0d7cb00c11 00007f0d7cac558a 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772
0806 03:52:38.117336 (+ 600us) log.cc:842] Serialized 26 byte log entry
0806 03:52:38.276338 (+159002us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:38.276386 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:38.276657 (+ 271us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:41.256355 (+2979698us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b44000232cc. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d799e1f11 00007f0d79aafa95 00007f0d79ab04b2 00007f0d7c758511 00007f0d7c753e1a 00007f0d7c747bee 00007f0d7c746f36 00007f0d7c746b58 00007f0d7c750551 00007f0d7c750509 00007f0d7c750499
0806 03:52:41.493562 (+237207us) raft_consensus.cc:1671] finished
0806 03:52:41.493590 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:41.493739 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":420,"raft.queue_time_us":79686,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":11941,"spinlock_wait_cycles":12198528,"thread_start_us":452003,"threads_started":2,"wal-append.queue_time_us":389170}
I20260806 03:52:41.530893 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:41.810380 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 48299.3us
I20260806 03:52:41.751701 3591 rpcz_store.cc:276] 0806 03:52:39.079461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:39.079627 (+ 166us) service_pool.cc:227] Handling call
0806 03:52:39.079995 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:39.080276 (+ 281us) raft_consensus.cc:1520] Early marking committed up to index 8
0806 03:52:39.377523 (+297247us) log.cc:842] Serialized 14 byte log entry
0806 03:52:41.363406 (+1985883us) spinlock_profiling.cc:243] Waited 3.48 ms on lock 0x7b180014a4c0. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:41.743724 (+380318us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:41.743792 (+ 68us) raft_consensus.cc:1631] Marking committed up to 8
0806 03:52:41.743847 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:41.743921 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:41.744075 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2745,"prepare.queue_time_us":830462,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":55,"raft.queue_time_us":486950,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":31200,"spinlock_wait_cycles":8004864,"thread_start_us":2640174,"threads_started":2}
I20260806 03:52:41.872170 2578 raft_consensus.cc:1226] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260806 03:52:41.810076 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 362651us
W20260806 03:52:41.829911 2429 leader_election.cc:334] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 3.359s (SENT)
I20260806 03:52:41.881171 2480 raft_consensus.cc:3037] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:41.881657 2429 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:41.879199 2574 rpcz_store.cc:276] 0806 03:52:40.713050 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:41.490642 (+777592us) service_pool.cc:227] Handling call
0806 03:52:41.597070 (+106428us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:41.769446 (+172376us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:41.769868 (+ 422us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260806 03:52:41.869972 3547 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 403644us
I20260806 03:52:41.877842 2579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:44194 (request call id 2451) took 1101 ms. Trace:
I20260806 03:52:41.482618 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:41.887921 2478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:41.888836 2575 raft_consensus.cc:1226] T 0d63bf35a2e44ee881530227ba8e7e99 P fe3d17f4ad984b9088a268d58d87469c [term 5 LEADER]: Rejecting Update request from peer 608e57d72aae496aaf8613622153aa0c for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260806 03:52:41.887826 2579 rpcz_store.cc:276] 0806 03:52:40.776961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:41.460682 (+683721us) service_pool.cc:227] Handling call
0806 03:52:41.877736 (+417054us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:41.892753 2578 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1221) took 1155 ms. Trace:
I20260806 03:52:41.893571 2578 rpcz_store.cc:276] 0806 03:52:40.722129 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:41.833313 (+1111184us) service_pool.cc:227] Handling call
0806 03:52:41.833876 (+ 563us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:41.872687 (+ 38811us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:41.877363 (+ 4676us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:41.895229 2578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:41.492616 7376 raft_consensus.cc:3032] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 9 LEADER]: Stepping down as leader of term 9
I20260806 03:52:41.898685 2575 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44204 (request call id 1218) took 1184 ms. Trace:
I20260806 03:52:41.901906 2575 rpcz_store.cc:276] 0806 03:52:40.713854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:41.810714 (+1096860us) service_pool.cc:227] Handling call
0806 03:52:41.888532 (+ 77818us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:41.898349 (+ 9817us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:41.898611 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:41.614813 7271 leader_election.cc:288] T 95f6f3de31d442cfa812406b89ee2fe3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:41.905201 2534 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:41.907827 2429 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 77996.5us
I20260806 03:52:41.875373 7242 raft_consensus.cc:2781] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:52:41.870509 7163 raft_consensus.cc:482] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:41.809224 7352 raft_consensus.cc:504] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.913235 7383 raft_consensus.cc:482] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:41.460063 7229 raft_consensus.cc:504] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.871531 3584 raft_consensus.cc:2454] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:41.871522 7243 raft_consensus.cc:482] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:41.917132 2577 raft_consensus.cc:3037] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:41.817621 7335 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.878504 7350 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:41.761875 7373 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:41.919030 7271 raft_consensus.cc:482] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:41.921825 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:41.924614 2484 raft_consensus.cc:2396] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260806 03:52:41.889706 7369 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:41.925868 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:41.903435 6770 raft_consensus.cc:2726] T bd8a4475fdfd488b8b2a5456043cfbc5 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:41.901729 7376 raft_consensus.cc:729] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:41.873251 7308 raft_consensus.cc:504] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.932194 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "507574ca8b05468888fa8465ccdb2f80" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:41.822587 7113 raft_consensus.cc:504] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.917311 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:41.817345 7374 raft_consensus.cc:504] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.924861 7242 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:41.940788 3588 raft_consensus.cc:2396] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 9 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260806 03:52:41.958308 2480 raft_consensus.cc:2454] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:41.960193 7367 consensus_queue.cc:1035] T a930c23072484f7ea5974173f5843bfe P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.030s
I20260806 03:52:41.961182 7221 consensus_queue.cc:1046] T d7a939d2320d4a478b95733e1579e2b3 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.056s
I20260806 03:52:41.948318 7373 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.961431 6384 consensus_queue.cc:1046] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260806 03:52:41.949361 7185 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:41.983191 2483 raft_consensus.cc:2454] T f1d501df3fe34842812664d2f3ec81fb P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
I20260806 03:52:41.489675 7165 raft_consensus.cc:504] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } attrs { replace: true } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: NON_VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: true } }
I20260806 03:52:41.809724 7353 raft_consensus.cc:504] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.970481 7148 raft_consensus.cc:3037] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 5 FOLLOWER]: Advancing to term 6
I20260806 03:52:41.930467 7203 consensus_queue.cc:1035] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260806 03:52:41.981500 3545 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:41.810257 7366 raft_consensus.cc:504] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.933786 7163 raft_consensus.cc:504] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:41.973680 7005 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:42.004398 2577 raft_consensus.cc:2454] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:42.006667 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da90827d7f0646f5bb848777586aae32" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:42.032306 2480 raft_consensus.cc:3037] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:41.943245 7383 raft_consensus.cc:504] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:41.951047 7243 raft_consensus.cc:504] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:41.974254 7271 raft_consensus.cc:504] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:41.999993 7242 raft_consensus.cc:3037] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:52:41.986774 7369 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.033078 2483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
W20260806 03:52:42.016477 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 90236.5us
I20260806 03:52:42.036314 2478 raft_consensus.cc:2379] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 10: Already voted for candidate 608e57d72aae496aaf8613622153aa0c in this term.
I20260806 03:52:42.077090 7243 leader_election.cc:288] T 3fb2162a24b741ebbc9732b674259c2d P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.080925 2581 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
W20260806 03:52:42.042235 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 137190us
I20260806 03:52:42.042488 3592 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260806 03:52:42.027274 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 45916.3us
I20260806 03:52:42.041325 2480 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:42.015805 7185 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.029234 2436 leader_election.cc:302] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.018481 6770 raft_consensus.cc:482] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:42.040681 7005 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.053714 7234 leader_election.cc:288] T 1f9fc024596a478e80a8be24774725e5 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:42.051609 7113 leader_election.cc:288] T f5efa489aaa94f9988b77e5efc032e5e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.055987 7323 leader_election.cc:288] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.032642 7072 consensus_queue.cc:1046] T 1f9fc024596a478e80a8be24774725e5 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260806 03:52:42.057125 6490 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.058277 7379 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:42.059747 7385 consensus_queue.cc:1035] T fb91ba465b9845129828f07c4d452970 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260806 03:52:42.065462 7374 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.066094 7163 leader_election.cc:288] T 2b2fa42f901d4b37bbfd3baccb897807 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.067185 7308 leader_election.cc:288] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.068233 7221 raft_consensus.cc:3032] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:42.134994 7221 raft_consensus.cc:729] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:42.100672 7185 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
W20260806 03:52:42.085870 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 56727.7us
I20260806 03:52:42.139837 2436 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.102105 3590 raft_consensus.cc:2454] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8.
I20260806 03:52:42.106647 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:42.055086 6896 raft_consensus.cc:3037] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 2 FOLLOWER]: Advancing to term 3
I20260806 03:52:42.110445 2534 leader_election.cc:302] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.063678 7271 leader_election.cc:288] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.145730 7229 leader_election.cc:288] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.146457 7381 consensus_queue.cc:1035] T c4f2c284701f44e495902166ab19b186 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260806 03:52:42.147650 2538 leader_election.cc:302] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.148518 7298 leader_election.cc:288] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.148710 7203 raft_consensus.cc:2781] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:42.149529 7203 raft_consensus.cc:482] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.103327 7376 consensus_queue.cc:260] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P cf6042e85bef47c4aac9d064d91501d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:42.150315 7203 raft_consensus.cc:3037] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:42.150754 2534 leader_election.cc:302] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c
I20260806 03:52:42.152420 2534 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.153494 2534 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:42.059607 7383 leader_election.cc:288] T dad99dac270543b19675707d75149e78 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.063846 7214 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.110281 6770 raft_consensus.cc:504] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.105402 7243 raft_consensus.cc:2781] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:42.130630 7379 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:42.143154 6490 raft_consensus.cc:2781] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:42.143285 7221 consensus_queue.cc:260] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.144418 7367 consensus_queue.cc:1046] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260806 03:52:42.157598 2534 leader_election.cc:302] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:42.204720 6384 raft_consensus.cc:3032] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:42.205300 6384 raft_consensus.cc:729] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:42.205734 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 48225.5us
I20260806 03:52:42.207301 2534 leader_election.cc:302] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:42.141585 2480 raft_consensus.cc:2454] T 04b1317a33c34953b84b5a8c6598368f P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 9.
I20260806 03:52:42.208954 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:42.209172 2376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2494) took 3124 ms. Trace:
I20260806 03:52:42.117782 7153 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.191529 7214 raft_consensus.cc:2781] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:52:42.211862 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2390) took 2073 ms. Trace:
I20260806 03:52:42.212338 7214 raft_consensus.cc:482] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:42.212821 2483 raft_consensus.cc:3037] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Advancing to term 7
I20260806 03:52:42.213063 7214 raft_consensus.cc:3037] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:52:42.214659 2482 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2382) took 2255 ms. Trace:
I20260806 03:52:42.176931 2436 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters: fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.215075 2482 rpcz_store.cc:276] 0806 03:52:39.958841 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:39.980645 (+ 21804us) service_pool.cc:227] Handling call
0806 03:52:40.003626 (+ 22981us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:40.270910 (+267284us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.214379 (+1943469us) spinlock_profiling.cc:243] Waited 178 ms on lock 0x7b540019ee54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:42.214589 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":409239808}
I20260806 03:52:42.201970 7221 raft_consensus.cc:3037] T 97ea60c38c204c989b4ca800e8478f18 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:42.206736 6384 consensus_queue.cc:260] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.227299 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95f6f3de31d442cfa812406b89ee2fe3" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.175642 7298 raft_consensus.cc:2781] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:52:42.180382 7381 raft_consensus.cc:2781] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:42.216603 7153 raft_consensus.cc:2741] T da90827d7f0646f5bb848777586aae32 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260806 03:52:42.211139 7397 raft_consensus.cc:2726] T 750d734a677a4861bca565b69d29396e P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:42.158097 2538 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.215998 7236 consensus_queue.cc:1035] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260806 03:52:42.216738 2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2499) took 2306 ms. Trace:
I20260806 03:52:42.190670 6490 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.209633 2480 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:52:42.218384 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 2.16133e+06us
I20260806 03:52:42.247578 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.248082 7355 leader_election.cc:288] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.248611 3590 raft_consensus.cc:2396] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 7.
I20260806 03:52:42.138338 7223 leader_election.cc:288] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:42.251379 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.242094 7381 raft_consensus.cc:686] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [term 6 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:42.219473 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:42.254488 2482 raft_consensus.cc:2454] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:42.212445 2474 rpcz_store.cc:276] 0806 03:52:40.138794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:40.641286 (+502492us) service_pool.cc:227] Handling call
0806 03:52:40.685065 (+ 43779us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:40.785461 (+100396us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:40.785520 (+ 59us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:40.785602 (+ 82us) raft_consensus.cc:1631] Marking committed up to 5
0806 03:52:40.785652 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.211573 (+1425921us) spinlock_profiling.cc:243] Waited 100 ms on lock 0x7b5400350054. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:42.211633 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:42.211789 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":230037888}
I20260806 03:52:42.162092 7234 consensus_queue.cc:1046] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260806 03:52:42.225814 3591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.210371 6770 leader_election.cc:288] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:42.234728 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 2.09717e+06us
I20260806 03:52:42.235145 3593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.235896 2373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 2490) took 3257 ms. Trace:
I20260806 03:52:42.187247 7243 raft_consensus.cc:482] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.236865 7335 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.188663 7165 leader_election.cc:288] T 14fa621f65da481d96d569096ded0a35 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 10 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.236568 2534 leader_election.cc:302] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
W20260806 03:52:42.265621 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 107785us
I20260806 03:52:42.176411 7072 raft_consensus.cc:2781] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election won for term 6
I20260806 03:52:42.255990 3547 leader_election.cc:302] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
I20260806 03:52:42.228890 7290 raft_consensus.cc:2726] T b91e8fd3fd1340d2bd8db09b77bd7b0f P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:42.255766 6490 raft_consensus.cc:3037] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:42.190045 7383 consensus_queue.cc:1046] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.049s
I20260806 03:52:42.233157 7298 raft_consensus.cc:482] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.209649 2376 rpcz_store.cc:276] 0806 03:52:39.085300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:39.085466 (+ 166us) service_pool.cc:227] Handling call
0806 03:52:39.099041 (+ 13575us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:39.109733 (+ 10692us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:39.109782 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:39.849870 (+740088us) log.cc:842] Serialized 26 byte log entry
0806 03:52:39.961599 (+111729us) raft_consensus.cc:1631] Marking committed up to 4
0806 03:52:39.995170 (+ 33571us) log.cc:842] Serialized 14 byte log entry
0806 03:52:41.493548 (+1498378us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b1800440f20. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b9d638 00007f0d79ba0b11 00007f0d79ba0ac9 00007f0d79ba0a59 00007f0d79ba0a21 00007f0d79b9fd1d 00007f0d82e01204 00007f0d82e01039 00007f0d79b7fdc6
0806 03:52:42.208515 (+714967us) spinlock_profiling.cc:243] Waited 41.3 ms on lock 0x7b1800440f20. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d79b144ca 00007f0d831f5f99 00007f0d799d8990 00007f0d79b9a864 00007f0d79b9a155 00007f0d7cabd3d0 00007f0d7cacf576 00007f0d7cae00d7 00007f0d7cae0058
0806 03:52:42.208755 (+ 240us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.208856 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:42.208911 (+ 55us) raft_consensus.cc:1671] finished
0806 03:52:42.208926 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:42.209097 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":737494,"prepare.queue_time_us":36257,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":58,"raft.queue_time_us":10247,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":6842,"spinlock_wait_cycles":132855808,"thread_start_us":111232,"threads_started":1,"wal-append.queue_time_us":189160}
I20260806 03:52:42.260936 2373 rpcz_store.cc:276] 0806 03:52:38.978401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.978636 (+ 235us) service_pool.cc:227] Handling call
0806 03:52:39.015444 (+ 36808us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:39.015710 (+ 266us) raft_consensus.cc:1520] Early marking committed up to index 7
0806 03:52:39.306475 (+290765us) log.cc:842] Serialized 14 byte log entry
0806 03:52:42.235489 (+2929014us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:42.235551 (+ 62us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:42.235602 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.235693 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:42.235828 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1931686,"prepare.queue_time_us":116418,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":61,"raft.queue_time_us":1937519,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":11680,"spinlock_wait_cycles":172454144,"thread_start_us":1270705,"threads_started":2,"wal-append.queue_time_us":1693936}
I20260806 03:52:42.241667 2379 rpcz_store.cc:276] 0806 03:52:39.909955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:39.910107 (+ 152us) service_pool.cc:227] Handling call
0806 03:52:39.928265 (+ 18158us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:40.011897 (+ 83632us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.216413 (+2204516us) spinlock_profiling.cc:243] Waited 77.7 ms on lock 0x7b5400012a54. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac5f51 00007f0d7cac499e 00007f0d81a062e9 00007f0d7b0ec684 00007f0d7b0ec612 00007f0d7b0ec541 00007f0d7b0ec4bc 00007f0d7b0eb772 00007f0d7aeef94c 00007f0d7aeeed86
0806 03:52:42.216665 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":178655488}
I20260806 03:52:42.260210 7337 raft_consensus.cc:482] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:42.230687 7367 raft_consensus.cc:2781] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260806 03:52:42.273313 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:42.284363 2370 raft_consensus.cc:3037] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:42.204692 7401 raft_consensus.cc:2781] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:42.284919 7401 raft_consensus.cc:482] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.285346 7401 raft_consensus.cc:3037] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:42.285828 7234 raft_consensus.cc:3032] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:42.275633 2375 raft_consensus.cc:1261] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Refusing update from remote peer cf6042e85bef47c4aac9d064d91501d1: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260806 03:52:42.256806 7352 leader_election.cc:288] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.308756 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
W20260806 03:52:42.299960 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 63521.9us
I20260806 03:52:42.309656 2376 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:42.265215 7243 raft_consensus.cc:3037] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:42.277264 2474 raft_consensus.cc:1226] T 6364534d89cb43079c8dafda6b5b8875 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:42.311190 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2395) took 1618 ms. Trace:
I20260806 03:52:42.268539 6384 raft_consensus.cc:3037] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:42.311887 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:42.311553 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:42.277545 7223 raft_consensus.cc:2781] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:42.269408 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0cb0e9b9f35b4cd488afa1ae281597bc" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:42.282721 2576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1d501df3fe34842812664d2f3ec81fb" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.283007 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c4f2c284701f44e495902166ab19b186" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.266530 2332 leader_election.cc:302] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.286738 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db383c38ecd45f2b2cb8c46ff360638" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.316162 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:42.283430 2538 leader_election.cc:302] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.244359 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5efa489aaa94f9988b77e5efc032e5e" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.279907 7229 raft_consensus.cc:2781] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260806 03:52:42.311540 2474 rpcz_store.cc:276] 0806 03:52:40.692986 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:42.259360 (+1566374us) service_pool.cc:227] Handling call
0806 03:52:42.277085 (+ 17725us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:42.310897 (+ 33812us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.311121 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:42.312732 2534 leader_election.cc:302] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.282649 7298 raft_consensus.cc:3037] T fb91ba465b9845129828f07c4d452970 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Advancing to term 11
W20260806 03:52:42.257709 7238 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.667s user 0.035s sys 0.052s
I20260806 03:52:42.323035 2483 raft_consensus.cc:2454] T 223d3e4f95dc4407a7abb4d09b0cceb7 P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:42.324540 3590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.291652 2480 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:42.286578 7367 raft_consensus.cc:482] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.337854 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:42.339015 2376 raft_consensus.cc:2454] T 7901535e3d4042389b2a8853a881ff1d P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 7.
I20260806 03:52:42.339859 2375 raft_consensus.cc:1226] T 78ff87b2196e4600aed5d8442f9b9bdd P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 5. Current term is 6. Ops: []
I20260806 03:52:42.292393 7234 raft_consensus.cc:729] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:42.308568 7383 consensus_queue.cc:1225] T 7724acebda824fbcbf3d5bbc6a4b820c P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260806 03:52:42.355945 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.340634 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bcedcc1f7e94405bbfa0a2d56aa30fb" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:42.337127 6770 raft_consensus.cc:482] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:42.337419 7397 consensus_queue.cc:1035] T 8390916b6ac84b59b3943bf835a36a4c P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260806 03:52:42.286513 7373 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.364269 2574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b082125feb462d9ba064ed2acf4dfd" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:42.364418 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:42.328500 7337 raft_consensus.cc:504] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:42.309103 7072 raft_consensus.cc:686] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [term 6 LEADER]: Becoming Leader. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:42.366483 3594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:36484 (request call id 359) took 2396 ms. Trace:
I20260806 03:52:42.394713 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:42.386925 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:42.395167 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32d01287577744949bfebb63c59f3db6" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:42.395978 2474 raft_consensus.cc:2362] T 32d01287577744949bfebb63c59f3db6 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6. Re-sending same reply.
W20260806 03:52:42.396938 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 84361.6us
I20260806 03:52:42.369264 2379 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:42.369557 2480 raft_consensus.cc:3037] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:42.398123 2379 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:42.371714 2370 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 8.
I20260806 03:52:42.367200 2537 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.336165 7229 raft_consensus.cc:482] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Starting leader election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:42.390738 7413 raft_consensus.cc:2781] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:42.407176 2436 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.408162 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:42.409025 2579 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
W20260806 03:52:42.360204 7238 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.774s user 0.036s sys 0.056s
I20260806 03:52:42.253443 7381 consensus_queue.cc:237] T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.330287 7258 raft_consensus.cc:2781] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260806 03:52:42.330638 7355 raft_consensus.cc:2781] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:42.393335 2482 raft_consensus.cc:2379] T 47ed8dc77b144ba08169a0090b787536 P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c in current term 9: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
W20260806 03:52:42.394276 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 133419us
I20260806 03:52:42.423864 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c"
I20260806 03:52:42.424607 2474 raft_consensus.cc:2362] T d48e3c6f820045128d5e9bad24b98699 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 8. Re-sending same reply.
I20260806 03:52:42.424894 2332 leader_election.cc:302] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.336715 7290 raft_consensus.cc:2726] T dad99dac270543b19675707d75149e78 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260806 03:52:42.331032 7204 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:42.392859 7366 leader_election.cc:288] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.395133 3594 rpcz_store.cc:276] 0806 03:52:39.969879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:39.970025 (+ 146us) service_pool.cc:227] Handling call
0806 03:52:39.996705 (+ 26680us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:39.997019 (+ 314us) raft_consensus.cc:1520] Early marking committed up to index 6
0806 03:52:40.152725 (+155706us) log.cc:842] Serialized 14 byte log entry
0806 03:52:42.365858 (+2213133us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b180040fa80. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d82a3c8b1 00007f0d82a49661 00007f0d7c75482c 00007f0d79b7f9dc 00007f0d829b6a39 00007f0d79b9c8bb 00007f0d79b9acbf 00007f0d79b9a155 00007f0d7cabd3d0 00007f0d7cacf576 00007f0d7cae00d7
0806 03:52:42.365987 (+ 129us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0806 03:52:42.366078 (+ 91us) raft_consensus.cc:1631] Marking committed up to 6
0806 03:52:42.366151 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.366253 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:42.366406 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16395,"prepare.queue_time_us":793575,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":40836,"raft.queue_time_us":601330,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":19039,"spinlock_wait_cycles":151980416,"thread_start_us":949778,"threads_started":3,"wal-append.queue_time_us":1256123}
W20260806 03:52:42.390048 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 58050.1us
I20260806 03:52:42.358407 7367 raft_consensus.cc:3037] T f327872ee26848bbaf04c03cc38edd54 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Advancing to term 11
I20260806 03:52:42.338142 7223 raft_consensus.cc:686] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:42.435647 7405 raft_consensus.cc:482] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
I20260806 03:52:42.444772 7409 raft_consensus.cc:482] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:42.445724 7409 raft_consensus.cc:504] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.380805 7153 consensus_queue.cc:1046] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260806 03:52:42.405040 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.448346 7336 raft_consensus.cc:2726] T 2ccb469db1664d3e941e7a360d048af2 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260806 03:52:42.448614 7366 raft_consensus.cc:3032] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 LEADER]: Stepping down as leader of term 8
I20260806 03:52:42.449131 7366 raft_consensus.cc:729] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:42.386762 7353 leader_election.cc:288] T 2401ce53befd43c6a455e89c5df49a28 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.428306 7355 raft_consensus.cc:482] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.419546 2331 leader_election.cc:302] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.410589 7229 raft_consensus.cc:3037] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:42.419996 7258 raft_consensus.cc:482] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
I20260806 03:52:42.457278 7258 raft_consensus.cc:3037] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 11 FOLLOWER]: Advancing to term 12
I20260806 03:52:42.330734 7352 raft_consensus.cc:482] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:42.430809 3587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.413184 7236 raft_consensus.cc:2781] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260806 03:52:42.366796 7224 raft_consensus.cc:482] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:42.417634 7271 raft_consensus.cc:2726] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:42.396299 7383 consensus_queue.cc:1046] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260806 03:52:42.392371 6770 raft_consensus.cc:504] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:42.425789 2436 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.183.3:46407 timed out after 2.071s (SENT) [suppressed 26 similar messages]
I20260806 03:52:42.458793 7355 raft_consensus.cc:3037] T 04b1317a33c34953b84b5a8c6598368f P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:42.450893 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5489408bdcc042f3919eae139ea82e7a" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:42.495641 2482 raft_consensus.cc:2454] T 5489408bdcc042f3919eae139ea82e7a P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 7.
I20260806 03:52:42.496769 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2401ce53befd43c6a455e89c5df49a28" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:42.497300 2436 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 71560.9us
I20260806 03:52:42.497715 2482 raft_consensus.cc:2396] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cf6042e85bef47c4aac9d064d91501d1 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260806 03:52:42.427646 7197 raft_consensus.cc:482] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:42.432642 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47ed8dc77b144ba08169a0090b787536" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:42.509079 2370 raft_consensus.cc:3037] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:42.466270 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.420603 7413 raft_consensus.cc:482] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260806 03:52:42.467514 2331 connection.cc:232] RPC call timeout handler was delayed by -1.2387918690219522s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1531s for a response
I20260806 03:52:42.467366 3545 leader_election.cc:302] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.463938 7290 raft_consensus.cc:2726] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P dff94100ffc5485f92cbe6a85addf0b2 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260806 03:52:42.476965 7379 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.469372 7403 raft_consensus.cc:2781] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Leader election won for term 8
I20260806 03:52:42.470594 7238 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:52:42.472198 5510 consensus_queue.cc:1159] T 7901535e3d4042389b2a8853a881ff1d P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "cf6042e85bef47c4aac9d064d91501d1" responder_term: 8 status { last_received { term: 7 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260806 03:52:42.474234 3584 raft_consensus.cc:1261] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Refusing update from remote peer dff94100ffc5485f92cbe6a85addf0b2: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260806 03:52:42.479001 3542 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: 608e57d72aae496aaf8613622153aa0c, dff94100ffc5485f92cbe6a85addf0b2
I20260806 03:52:42.479405 2480 raft_consensus.cc:2454] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 10.
I20260806 03:52:42.542930 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa9a0499fc04e10aefa0602a7adbbb1" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:42.543574 2480 raft_consensus.cc:1799] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Handling vote request from an unknown peer fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.544154 2480 raft_consensus.cc:2396] T dfa9a0499fc04e10aefa0602a7adbbb1 P 608e57d72aae496aaf8613622153aa0c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe3d17f4ad984b9088a268d58d87469c for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20260806 03:52:42.442255 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.451445 2474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:42.499179 2579 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:42.499282 2538 leader_election.cc:302] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.546595 2474 raft_consensus.cc:2454] T 4fed83d97aac42e4b3e69249098b6111 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3d17f4ad984b9088a268d58d87469c in term 6.
W20260806 03:52:42.547046 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 47870.7us
W20260806 03:52:42.501924 7404 log.cc:924] Time spent T 3fb2162a24b741ebbc9732b674259c2d P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.190s user 0.000s sys 0.001s
W20260806 03:52:42.503263 7349 log.cc:924] Time spent T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 1.300s user 0.000s sys 0.001s
I20260806 03:52:42.480679 7236 raft_consensus.cc:482] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Starting leader election (detected failure of leader dff94100ffc5485f92cbe6a85addf0b2)
I20260806 03:52:39.861146 2479 rpcz_store.cc:276] 0806 03:52:38.665410 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.665945 (+ 535us) service_pool.cc:227] Handling call
0806 03:52:38.683469 (+ 17524us) raft_consensus.cc:1457] Updating replica for 1 ops
0806 03:52:38.714778 (+ 31309us) raft_consensus.cc:1520] Early marking committed up to index 0
0806 03:52:38.714804 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0806 03:52:38.746562 (+ 31758us) log.cc:842] Serialized 26 byte log entry
0806 03:52:38.993132 (+246570us) raft_consensus.cc:1631] Marking committed up to 7
0806 03:52:39.193525 (+200393us) log.cc:842] Serialized 14 byte log entry
0806 03:52:39.844079 (+650554us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:39.844166 (+ 87us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0806 03:52:39.844210 (+ 44us) raft_consensus.cc:1671] finished
0806 03:52:39.844222 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0806 03:52:39.844377 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18866,"prepare.queue_time_us":326190,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":74,"raft.queue_time_us":667132,"raft.run_cpu_time_us":367,"raft.run_wall_time_us":40700,"thread_start_us":221798,"threads_started":2,"wal-append.queue_time_us":105378}
I20260806 03:52:42.507596 7005 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.476156 7336 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader fe3d17f4ad984b9088a268d58d87469c)
I20260806 03:52:42.437314 7204 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:42.511677 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:42.468344 7375 raft_consensus.cc:482] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 608e57d72aae496aaf8613622153aa0c)
W20260806 03:52:42.517987 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 50714.1us
I20260806 03:52:42.519109 7197 raft_consensus.cc:504] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.520740 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7901535e3d4042389b2a8853a881ff1d" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.483264 7224 raft_consensus.cc:504] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.493839 7383 consensus_queue.cc:1225] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [LEADER]: Peer fe3d17f4ad984b9088a268d58d87469c log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260806 03:52:42.545059 2480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f9fc024596a478e80a8be24774725e5" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
W20260806 03:52:42.549798 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 70833.9us
I20260806 03:52:42.480337 7072 consensus_queue.cc:237] T 750d734a677a4861bca565b69d29396e P 608e57d72aae496aaf8613622153aa0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.569509 2482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "608e57d72aae496aaf8613622153aa0c" is_pre_election: true
I20260806 03:52:42.594038 3594 raft_consensus.cc:2379] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dff94100ffc5485f92cbe6a85addf0b2 in current term 8: Already voted for candidate cf6042e85bef47c4aac9d064d91501d1 in this term.
I20260806 03:52:42.563740 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dad99dac270543b19675707d75149e78" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.529588 7190 raft_consensus.cc:2726] T da90827d7f0646f5bb848777586aae32 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260806 03:52:42.572394 2370 raft_consensus.cc:2454] T 47ed8dc77b144ba08169a0090b787536 P dff94100ffc5485f92cbe6a85addf0b2 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6042e85bef47c4aac9d064d91501d1 in term 9.
I20260806 03:52:42.517232 7413 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:42.585709 3542 leader_election.cc:302] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
I20260806 03:52:42.481204 7352 raft_consensus.cc:504] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.569895 7236 raft_consensus.cc:3037] T e9f040dd271944b9bf08554a90afa940 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:42.591217 3545 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.487380 7374 consensus_queue.cc:1035] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260806 03:52:42.595755 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05e17a7474ab4829922b72cc624de950" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.607016 2579 raft_consensus.cc:2454] T 05e17a7474ab4829922b72cc624de950 P fe3d17f4ad984b9088a268d58d87469c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 4.
I20260806 03:52:42.599599 2537 leader_election.cc:302] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:42.601435 2480 raft_consensus.cc:1261] T 460a21786a7449039551dc0f1ffd04e2 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:42.589094 2474 raft_consensus.cc:1261] T 2401ce53befd43c6a455e89c5df49a28 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260806 03:52:42.449831 7234 consensus_queue.cc:260] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.608878 2474 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2396) took 1829 ms. Trace:
I20260806 03:52:42.613618 2474 rpcz_store.cc:276] 0806 03:52:40.779682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:42.562990 (+1783308us) service_pool.cc:227] Handling call
0806 03:52:42.588594 (+ 25604us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:42.608515 (+ 19921us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.608799 (+ 284us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:42.608655 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2400) took 1121 ms. Trace:
I20260806 03:52:42.614589 2480 rpcz_store.cc:276] 0806 03:52:41.487041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:42.584056 (+1097015us) service_pool.cc:227] Handling call
0806 03:52:42.584580 (+ 524us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:42.608365 (+ 23785us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.608591 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:42.616132 2479 raft_consensus.cc:1261] T b91e8fd3fd1340d2bd8db09b77bd7b0f P 608e57d72aae496aaf8613622153aa0c [term 7 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260806 03:52:42.616842 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2397) took 1732 ms. Trace:
I20260806 03:52:42.617168 2479 rpcz_store.cc:276] 0806 03:52:40.884639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:42.574021 (+1689382us) service_pool.cc:227] Handling call
0806 03:52:42.601722 (+ 27701us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:42.616540 (+ 14818us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.616771 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:42.575136 7403 raft_consensus.cc:686] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [term 8 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
I20260806 03:52:42.563203 7394 raft_consensus.cc:2726] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260806 03:52:42.563704 7238 raft_consensus.cc:504] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.625252 7238 leader_election.cc:288] T 05e17a7474ab4829922b72cc624de950 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.628804 2480 raft_consensus.cc:1261] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260806 03:52:42.629210 2480 pending_rounds.cc:76] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Aborting all ops after (but not including) 3
I20260806 03:52:42.629424 2480 pending_rounds.cc:97] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260806 03:52:42.629650 2480 raft_consensus.cc:2866] T d2ddf38aaea54266a0a9e6100c48fc0d P 608e57d72aae496aaf8613622153aa0c [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260806 03:52:42.630553 2480 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2403) took 1006 ms. Trace:
I20260806 03:52:42.630954 2480 rpcz_store.cc:276] 0806 03:52:41.624325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:42.628098 (+1003773us) service_pool.cc:227] Handling call
0806 03:52:42.628437 (+ 339us) raft_consensus.cc:1457] Updating replica for 0 ops
0806 03:52:42.630225 (+ 1788us) raft_consensus.cc:1683] Filling consensus response to leader.
0806 03:52:42.630489 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260806 03:52:42.632571 2482 raft_consensus.cc:3032] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Stepping down as leader of term 4
I20260806 03:52:42.537796 7229 raft_consensus.cc:504] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:42.646948 2436 leader_election.cc:302] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.633082 2482 raft_consensus.cc:729] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 608e57d72aae496aaf8613622153aa0c, State: Running, Role: LEADER
I20260806 03:52:42.596194 7383 raft_consensus.cc:482] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:42.612389 7241 raft_consensus.cc:2781] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260806 03:52:42.621677 7242 consensus_meta.cc:287] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.557s user 0.018s sys 0.000s
I20260806 03:52:42.665385 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2fa42f901d4b37bbfd3baccb897807" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.477126 7405 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.678709 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef53ef49663446658709042138ce12f7" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.690507 2436 leader_election.cc:302] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: cf6042e85bef47c4aac9d064d91501d1, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.692344 7335 raft_consensus.cc:2726] T ef53ef49663446658709042138ce12f7 P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:42.646106 7150 raft_consensus.cc:482] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:42.558775 7223 consensus_queue.cc:237] T da90827d7f0646f5bb848777586aae32 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:42.613611 7409 leader_election.cc:288] T d2ddf38aaea54266a0a9e6100c48fc0d P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.647687 7373 raft_consensus.cc:2781] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader election won for term 9
W20260806 03:52:42.698945 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 150819us
I20260806 03:52:42.577149 7336 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.654495 6896 raft_consensus.cc:2726] T 47ed8dc77b144ba08169a0090b787536 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260806 03:52:42.525085 7366 consensus_queue.cc:260] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.693344 7234 raft_consensus.cc:3037] T 27aa6efa6253470d9fb619f5536e8795 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:42.626139 7290 consensus_queue.cc:1035] T c210749b55e24fa0bb7b9c7cdf3cc6b0 P dff94100ffc5485f92cbe6a85addf0b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.087s
I20260806 03:52:42.650185 7353 raft_consensus.cc:2781] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260806 03:52:42.710600 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d48e3c6f820045128d5e9bad24b98699" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.704380 7136 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P 608e57d72aae496aaf8613622153aa0c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260806 03:52:42.696360 7349 log.cc:924] Time spent T 97ea60c38c204c989b4ca800e8478f18 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20260806 03:52:42.665186 2482 consensus_queue.cc:260] T 0d63bf35a2e44ee881530227ba8e7e99 P 608e57d72aae496aaf8613622153aa0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.708494 2538 leader_election.cc:302] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:42.687835 7241 raft_consensus.cc:482] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Starting leader election (detected failure of leader cf6042e85bef47c4aac9d064d91501d1)
W20260806 03:52:42.710000 7185 consensus_meta.cc:287] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.527s user 0.009s sys 0.005s
W20260806 03:52:42.742244 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 124676us
I20260806 03:52:42.684283 7153 raft_consensus.cc:2781] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260806 03:52:42.752801 2538 leader_election.cc:302] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c; no voters:
I20260806 03:52:42.689301 7242 raft_consensus.cc:504] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.787756 2538 leader_election.cc:302] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1
I20260806 03:52:42.667670 7383 raft_consensus.cc:504] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:42.757845 3545 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 166775us
W20260806 03:52:42.751586 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.3:46407, user_credentials={real_user=slave}} blocked reactor thread for 136130us
I20260806 03:52:42.755512 3542 leader_election.cc:302] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:42.781100 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a930c23072484f7ea5974173f5843bfe" candidate_uuid: "cf6042e85bef47c4aac9d064d91501d1" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.756532 7241 raft_consensus.cc:3037] T 5489408bdcc042f3919eae139ea82e7a P fe3d17f4ad984b9088a268d58d87469c [term 7 FOLLOWER]: Advancing to term 8
I20260806 03:52:42.726948 7337 leader_election.cc:288] T 39115846d5144463b3a3a8e09e8b4500 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.603526 7375 raft_consensus.cc:504] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
I20260806 03:52:42.743048 7353 raft_consensus.cc:482] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.702886 7150 raft_consensus.cc:504] T 84da474ba5944a7a84d7c76403b92a63 P fe3d17f4ad984b9088a268d58d87469c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.753964 7366 raft_consensus.cc:3037] T 0cb0e9b9f35b4cd488afa1ae281597bc P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:42.777731 2583 raft_consensus.cc:2454] T 6364534d89cb43079c8dafda6b5b8875 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 6.
W20260806 03:52:42.781735 7411 log.cc:924] Time spent T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.257s user 0.000s sys 0.002s
I20260806 03:52:42.791776 7412 raft_consensus.cc:2781] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:42.797446 7397 consensus_queue.cc:1035] T 460a21786a7449039551dc0f1ffd04e2 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260806 03:52:42.804870 7197 leader_election.cc:288] T c4f2c284701f44e495902166ab19b186 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.805234 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac682b8be46c4b6599610372a4c8c727" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c" is_pre_election: true
I20260806 03:52:42.809324 7229 leader_election.cc:288] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [CANDIDATE]: Term 8 election: Requested vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:42.816143 6490 consensus_meta.cc:287] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.529s user 0.017s sys 0.000s
W20260806 03:52:42.817236 7410 log.cc:924] Time spent T f327872ee26848bbaf04c03cc38edd54 P dff94100ffc5485f92cbe6a85addf0b2: Append to log took a long time: real 0.204s user 0.002s sys 0.000s
I20260806 03:52:42.767391 7232 raft_consensus.cc:2781] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election won for term 10
I20260806 03:52:42.670946 7394 consensus_queue.cc:1046] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } attrs { replace: true }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.117s
I20260806 03:52:42.785485 7153 raft_consensus.cc:482] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.792673 7151 raft_consensus.cc:2726] T 78ff87b2196e4600aed5d8442f9b9bdd P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260806 03:52:42.822247 6770 leader_election.cc:288] T ac682b8be46c4b6599610372a4c8c727 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.828086 7383 leader_election.cc:288] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.856801 7353 raft_consensus.cc:3037] T 7901535e3d4042389b2a8853a881ff1d P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Advancing to term 9
I20260806 03:52:42.774214 7185 raft_consensus.cc:504] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
I20260806 03:52:42.870175 7224 leader_election.cc:288] T 6364534d89cb43079c8dafda6b5b8875 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:42.818917 6915 consensus_queue.cc:1035] T 2401ce53befd43c6a455e89c5df49a28 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.093s
I20260806 03:52:42.958804 2579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "fe3d17f4ad984b9088a268d58d87469c"
I20260806 03:52:42.882872 7412 raft_consensus.cc:482] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 03:52:42.868676 3588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39115846d5144463b3a3a8e09e8b4500" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
I20260806 03:52:42.875918 7204 leader_election.cc:288] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), cf6042e85bef47c4aac9d064d91501d1 (127.0.183.4:33215)
I20260806 03:52:42.754235 7403 consensus_queue.cc:237] T d48e3c6f820045128d5e9bad24b98699 P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:42.931257 6384 consensus_meta.cc:287] T 01b082125feb462d9ba064ed2acf4dfd P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 0.608s user 0.012s sys 0.003s
I20260806 03:52:42.882917 7221 consensus_queue.cc:1035] T b91e8fd3fd1340d2bd8db09b77bd7b0f P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20260806 03:52:42.941062 7153 raft_consensus.cc:3037] T 05e17a7474ab4829922b72cc624de950 P 608e57d72aae496aaf8613622153aa0c [term 4 FOLLOWER]: Advancing to term 5
I20260806 03:52:43.051548 2579 raft_consensus.cc:3037] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 9 FOLLOWER]: Advancing to term 10
I20260806 03:52:42.943073 7409 raft_consensus.cc:1050] T 14fa621f65da481d96d569096ded0a35 P dff94100ffc5485f92cbe6a85addf0b2: attempting to promote NON_VOTER cf6042e85bef47c4aac9d064d91501d1 to VOTER
I20260806 03:52:43.068801 7336 leader_election.cc:288] T 84da474ba5944a7a84d7c76403b92a63 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
W20260806 03:52:43.048635 3585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:36474 (request call id 515) took 4531 ms (client timeout 1550 ms). Trace:
I20260806 03:52:42.963728 3584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d93b8d7e5d334d3dac131b9a4c49b103" candidate_uuid: "608e57d72aae496aaf8613622153aa0c" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:42.988306 7258 consensus_meta.cc:287] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.008s
I20260806 03:52:43.048005 7374 raft_consensus.cc:3032] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Stepping down as leader of term 5
I20260806 03:52:43.055470 7412 raft_consensus.cc:3037] T 4fed83d97aac42e4b3e69249098b6111 P fe3d17f4ad984b9088a268d58d87469c [term 6 FOLLOWER]: Advancing to term 7
W20260806 03:52:43.080353 7416 log.cc:924] Time spent T 32d01287577744949bfebb63c59f3db6 P fe3d17f4ad984b9088a268d58d87469c: Append to log took a long time: real 0.245s user 0.002s sys 0.000s
W20260806 03:52:42.985194 2332 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.008s sys 0.003s
W20260806 03:52:43.088243 2534 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.006s sys 0.005s
W20260806 03:52:43.146351 2332 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.008s sys 0.003s
I20260806 03:52:43.080821 3594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3ae6cf9eba9410d8307ed1da6a62ea6" candidate_uuid: "dff94100ffc5485f92cbe6a85addf0b2" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "cf6042e85bef47c4aac9d064d91501d1" is_pre_election: true
W20260806 03:52:43.093490 3542 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 338049us
I20260806 03:52:43.054793 7194 raft_consensus.cc:2726] T e9f040dd271944b9bf08554a90afa940 P cf6042e85bef47c4aac9d064d91501d1 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260806 03:52:42.961153 7232 raft_consensus.cc:686] T 9bcedcc1f7e94405bbfa0a2d56aa30fb P fe3d17f4ad984b9088a268d58d87469c [term 10 LEADER]: Becoming Leader. State: Replica: fe3d17f4ad984b9088a268d58d87469c, State: Running, Role: LEADER
W20260806 03:52:43.078898 2538 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 280201us
I20260806 03:52:43.117270 3584 raft_consensus.cc:2454] T d93b8d7e5d334d3dac131b9a4c49b103 P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 608e57d72aae496aaf8613622153aa0c in term 8.
I20260806 03:52:42.984259 7352 leader_election.cc:288] T 2ccb469db1664d3e941e7a360d048af2 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:43.119915 3591 raft_consensus.cc:1226] T a930c23072484f7ea5974173f5843bfe P cf6042e85bef47c4aac9d064d91501d1 [term 8 FOLLOWER]: Rejecting Update request from peer fe3d17f4ad984b9088a268d58d87469c for earlier term 7. Current term is 8. Ops: []
I20260806 03:52:42.932965 6490 raft_consensus.cc:504] T 1d90046b84bc4833ac8950bd501840d7 P fe3d17f4ad984b9088a268d58d87469c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "fe3d17f4ad984b9088a268d58d87469c" member_type: VOTER last_known_addr { host: "127.0.183.3" port: 46407 } }
W20260806 03:52:43.169004 2534 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.762s user 0.008s sys 0.006s
W20260806 03:52:43.190737 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 783502us
I20260806 03:52:43.167840 3594 raft_consensus.cc:2454] T f3ae6cf9eba9410d8307ed1da6a62ea6 P cf6042e85bef47c4aac9d064d91501d1 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dff94100ffc5485f92cbe6a85addf0b2 in term 6.
I20260806 03:52:43.043802 7373 raft_consensus.cc:686] T 47ed8dc77b144ba08169a0090b787536 P cf6042e85bef47c4aac9d064d91501d1 [term 9 LEADER]: Becoming Leader. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:43.177464 2479 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:54220 (request call id 2404) took 1552 ms. Trace:
I20260806 03:52:43.119818 7374 raft_consensus.cc:729] T 78ff87b2196e4600aed5d8442f9b9bdd P cf6042e85bef47c4aac9d064d91501d1 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: cf6042e85bef47c4aac9d064d91501d1, State: Running, Role: LEADER
I20260806 03:52:43.144428 7405 leader_election.cc:288] T 1d90046b84bc4833ac8950bd501840d7 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689), fe3d17f4ad984b9088a268d58d87469c (127.0.183.3:46407)
I20260806 03:52:43.193030 2331 leader_election.cc:302] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2; no voters:
W20260806 03:52:43.301074 2331 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.4:33215, user_credentials={real_user=slave}} blocked reactor thread for 108149us
W20260806 03:52:43.296370 2534 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.183.1:39643, user_credentials={real_user=slave}} blocked reactor thread for 104689us
I20260806 03:52:43.299101 2436 leader_election.cc:302] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c, cf6042e85bef47c4aac9d064d91501d1; no voters:
W20260806 03:52:43.080929 3585 rpcz_store.cc:269] 0806 03:52:38.511137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0806 03:52:38.513230 (+ 2093us) service_pool.cc:227] Handling call
0806 03:52:42.987722 (+4474492us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b5400732654. stack: 00007f0d79b6b033 00007f0d79b6af30 00007f0d794c9534 00007f0d831d58b4 00007f0d831d5849 00007f0d7ca4a471 00007f0d7cac92ae 00007f0d81a06c27 00007f0d7b0eec04 00007f0d7b0eeb92 00007f0d7b0eeac1 00007f0d7b0eea3c 00007f0d7b0edcf2 00007f0d7aeef94c 00007f0d7aeeed86 00007f0d7aeee72f
0806 03:52:43.029323 (+ 41601us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":289649920}
I20260806 03:52:43.177338 7242 leader_election.cc:288] T 8655b80017d7422082efde215e797b5b P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 11 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
W20260806 03:52:43.326330 2332 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.183.2:45689, user_credentials={real_user=slave}} blocked reactor thread for 901477us
I20260806 03:52:43.343816 2332 leader_election.cc:302] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:43.143684 7258 raft_consensus.cc:504] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "dff94100ffc5485f92cbe6a85addf0b2" member_type: VOTER last_known_addr { host: "127.0.183.1" port: 39643 } } peers { permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 } } peers { permanent_uuid: "cf6042e85bef47c4aac9d064d91501d1" member_type: VOTER last_known_addr { host: "127.0.183.4" port: 33215 } attrs { promote: false } }
W20260806 03:52:43.314901 7411 log.cc:924] Time spent T 39115846d5144463b3a3a8e09e8b4500 P cf6042e85bef47c4aac9d064d91501d1: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20260806 03:52:43.348155 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ead4b99d10d24d13ba7f07dee24f79e3" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 9 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:43.360966 7405 raft_consensus.cc:2726] T 32d01287577744949bfebb63c59f3db6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260806 03:52:43.361462 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c210749b55e24fa0bb7b9c7cdf3cc6b0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:43.362627 2332 leader_election.cc:302] T bd8a4475fdfd488b8b2a5456043cfbc5 P dff94100ffc5485f92cbe6a85addf0b2 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dff94100ffc5485f92cbe6a85addf0b2; no voters: 608e57d72aae496aaf8613622153aa0c, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:43.316455 7283 raft_consensus.cc:2781] T f3ae6cf9eba9410d8307ed1da6a62ea6 P dff94100ffc5485f92cbe6a85addf0b2 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260806 03:52:43.368505 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9f040dd271944b9bf08554a90afa940" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:43.320519 6770 raft_consensus.cc:2781] T d93b8d7e5d334d3dac131b9a4c49b103 P 608e57d72aae496aaf8613622153aa0c [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260806 03:52:43.307585 7203 consensus_meta.cc:287] T f1d501df3fe34842812664d2f3ec81fb P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.153s user 0.019s sys 0.016s
I20260806 03:52:43.353332 2370 raft_consensus.cc:1261] T 460a21786a7449039551dc0f1ffd04e2 P dff94100ffc5485f92cbe6a85addf0b2 [term 8 FOLLOWER]: Refusing update from remote peer fe3d17f4ad984b9088a268d58d87469c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260806 03:52:43.353678 7185 leader_election.cc:288] T 6796c2cfd90d4dc9ab9759b7845f0fd0 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 8 election: Requested vote from peers dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643), 608e57d72aae496aaf8613622153aa0c (127.0.183.2:45689)
I20260806 03:52:43.349624 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14fa621f65da481d96d569096ded0a35" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:43.379787 2372 raft_consensus.cc:1226] T 507574ca8b05468888fa8465ccdb2f80 P dff94100ffc5485f92cbe6a85addf0b2 [term 12 FOLLOWER]: Rejecting Update request from peer cf6042e85bef47c4aac9d064d91501d1 for earlier term 11. Current term is 12. Ops: []
I20260806 03:52:43.362684 2374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fed83d97aac42e4b3e69249098b6111" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:43.358990 7352 raft_consensus.cc:482] T 8655b80017d7422082efde215e797b5b P 608e57d72aae496aaf8613622153aa0c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 03:52:43.191370 6896 consensus_queue.cc:1035] T d2ddf38aaea54266a0a9e6100c48fc0d P fe3d17f4ad984b9088a268d58d87469c [LEADER]: Connected to new peer: Peer: permanent_uuid: "608e57d72aae496aaf8613622153aa0c" member_type: VOTER last_known_addr { host: "127.0.183.2" port: 45689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.163s
W20260806 03:52:43.352749 2429 leader_election.cc:334] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer dff94100ffc5485f92cbe6a85addf0b2 (127.0.183.1:39643): Timed out: RequestConsensusVote RPC to 127.0.183.1:39643 timed out after 1.553s (SENT)
W20260806 03:52:43.347240 7401 consensus_meta.cc:287] T ead4b99d10d24d13ba7f07dee24f79e3 P cf6042e85bef47c4aac9d064d91501d1: Time spent flushing consensus metadata: real 1.061s user 0.012s sys 0.021s
I20260806 03:52:43.378037 2534 leader_election.cc:302] T ead4b99d10d24d13ba7f07dee24f79e3 P fe3d17f4ad984b9088a268d58d87469c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe3d17f4ad984b9088a268d58d87469c; no voters: cf6042e85bef47c4aac9d064d91501d1, dff94100ffc5485f92cbe6a85addf0b2
W20260806 03:52:43.365420 7214 consensus_meta.cc:287] T 2aa3fe3cb0444d6ea2df79f78a0d7543 P fe3d17f4ad984b9088a268d58d87469c: Time spent flushing consensus metadata: real 1.152s user 0.011s sys 0.015s
I20260806 03:52:43.384696 2375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78ff87b2196e4600aed5d8442f9b9bdd" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:43.386988 2429 leader_election.cc:302] T d7a939d2320d4a478b95733e1579e2b3 P 608e57d72aae496aaf8613622153aa0c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 608e57d72aae496aaf8613622153aa0c; no voters: dff94100ffc5485f92cbe6a85addf0b2, fe3d17f4ad984b9088a268d58d87469c
I20260806 03:52:43.393631 2370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04b1317a33c34953b84b5a8c6598368f" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2" is_pre_election: true
I20260806 03:52:43.399837 2373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6796c2cfd90d4dc9ab9759b7845f0fd0" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "dff94100ffc5485f92cbe6a85addf0b2"
I20260806 03:52:43.395128 2579 raft_consensus.cc:2396] T 14fa621f65da481d96d569096ded0a35 P fe3d17f4ad984b9088a268d58d87469c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 608e57d72aae496aaf8613622153aa0c for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260806 03:52:43.395604 2376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "223d3e4f95dc4407a7abb4d09b0cceb7" candidate_uuid: "fe3d17f4ad984b9088a268d58d87469c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "